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Executive  Summary 

Command  and  Control  is  recognized  as  a  critical  element  of  successful  military  operations  and  a  key 
aspect  of  Information  Age  transformation.  Until  recently,  however,  physics-dominated  issues  of  military 
operations,  rather  than  C2  ones,  have  been  the  primary,  almost  exclusive  focus  of  military  analysis  and 
assessments.  This,  coupled  with  the  inherent  complexity  of  C2  (which  involves  both  the  information  and 
cognitive  domains),  has  presented  the  assessment  community  with  challenges  that  are  less  well  researched 
and  understood  and  with  a  tool  kit  that  is  clearly  lacking.  The  NATO  Code  of  Best  Practice  for 
C2  Assessment  (COBP),  therefore,  has  been  developed  to  help  C2  analysts  and  decision  makers  deal 
with  these  new  Information  Age  assessment  challenges  so  that  they  can  improve  their  ability  to  take 
on  analyses  of  requirements,  analyses  of  alternatives,  research  into  new  C2  concepts  and  capabilities, 
and  support  real  world  operations. 

The  COBP  is  the  product  of  international  collaboration  drawing  together  the  operational  and  analytical 
experience  of  leading  military  and  civilian  defence  experts  from  across  the  NATO  nations.  The  COBP 
enhances  the  understanding  of  best  practice  and  outlines  a  structured  process  for  the  conduct  of 
operational  assessment  for  C2,  which  is  the  core  capability  of  Information  Age  defence  and  security. 
Use  of  the  COBP  will: 

•  Increase  the  likelihood  of  quality  products  being  : 

•  Complete; 

•  Relevant; 

•  Transparent; 

•  Credible; 

•  Authoritative; 

•  And  reduce  the  risk/cost  associated  with  producing  the  products. 

This  COBP  has  been  extended  from  the  initial  version  to  cover  issues  unique  to  challenges  related  to 
operations  other  than  war  (OOTW).  The  Code  is  designed  to  guide  both  decision  makers  and  analysts  by 
means  of  an  assessment  framework,  from  concept  development,  through  to  assessment  products  and  the 
dissemination  of  findings  and  conclusions.  To  do  this,  the  COBP  discusses  assessment  preparation  in  the 
form  of:  study  dynamics,  problem  formulation  and  possible  solution  strategies;  assessment:  measures  of 
merit,  scenarios,  human  and  organisational  issues,  data  and  tools;  risk  and  uncertainty;  and  the  full  range 
of  assessment  products. 
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Code  OTAN  des  meilleures  pratiques  pour 
revaluation  du  commandement  et  du  controle 

(RTO  TR-081  /  SAS-026) 

Synthese 

Le  commandement  et  controle  (C2)  est  considere  comme  I’un  des  elements  determinants  de  la  reussite  des 
operations  militaires,  ainsi  qu’un  aspect  cle  des  transformations  engendrees  par  I’ere  de  I’information. 
Toutefois,  il  y  a  peu  de  temps  encore,  les  analyses  et  evaluations  militaires  portaient  en  premier  lieu,  et 
dans  leur  quasi-totalite,  sur  les  aspects  principalement  physiques  des  operations  militaires  plutot  que  sur  le 
C2.  Cette  situation,  conjuguee  a  la  complexite  propre  au  C2  (qui  implique  tant  le  domaine  de  I’information 
que  le  domaine  cognitif),  a  mis  les  evaluateurs  devant  des  defis  qui,  faute  de  recherches,  sont  moins  bien 
compris  ,sans  compter  qu’ils  manquent  a  I’evidence,  les  outils  necessaires.  Un  Code  OTAN  des  meilleures 
pratiques  pour  revaluation  du  C2  (COBP)  a  done  ete  etabli  afin  d’ aider  les  analystes  et  decideurs  dans  ce 
domaine  a  faire  face  a  ces  nouvelles  difficultes  d’ evaluation  bees  a  Tere  de  P  information  de  faqon  a  ce 
qu’ils  puissent  ameliorer  leur  aptitude  a  effectuer  des  analyses  de  besoins,  des  analyses  des  diverses 
solutions  possibles  et  des  recherches  sur  de  nouveaux  concepts  et  capacites  C2,  ainsi  qu’a  soutenir  les 
operations  reelles. 

Le  COBP  est  le  resultat  d’une  collaboration  intemationale  mettant  en  common  Texperience  operationnelle 
et  analytique  d’eminents  experts  militaires  et  civils  du  secteur  de  la  defense  appartenant  a  tons  les  pays  de 
rOTAN.  Ce  code  facilite  la  comprehension  des  meilleures  pratiques  et  donne  les  grandes  lignes  d’un 
processus  structure  pour  revaluation  operationnelle  du  C2,  capacite  centrale  pour  la  defense  et  la  securite 
a  I’ere  de  I’information.  Les  aspects  C2  d’une  capacite  militaire  sont  difficiles  a  evaluer.  L’utilisation  du 
COBP  permettra  : 

•  d’accroitre  la  probabilite  d’obtenir  des  produits  de  qualite  ayant  I’avantage  d’etre  ; 

•  complets ; 

•  pertinents ; 

•  transparents  ; 

•  credibles ; 

•  faisant  autorite  ; 

•  et  de  reduire  les  risques/couts  lies  a  leur  elaboration. 

La  version  initiale  du  COBP  a  ete  developpee  de  maniere  a  couvrir  des  questions  specifiques  aux 
difficultes  bees  aux  operations  autres  que  celles  de  guerre  (OAQG).  Ce  code  est  destine  a  guider  tant  les 
decideurs  que  les  analystes,  par  le  biais  d’une  evaluation,  des  I’etape  de  I’etabbssement  du  concept, 
jusqu’a  celle  de  la  realisation  de  produits  d'evaluation  et  de  la  diffusion  des  conclusions.  A  cet  effet,  le 
COBP  porte  sur  la  preparation  des  evaluations  en  termes  de  dynamique  d’ etude,  de  formulation  de 
problemes  et  de  strategies  susceptibles  de  les  resoudre,  sur  revaluation  du  degre  d’interet,  de  scenarios,  de 
problemes  humains  et  organisationnels,  de  donnees  et  d’outils,  sur  les  risques  et  les  incertitudes,  ainsi  que 
sur  tout  I’eventail  des  produits  d’evaluation. 
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Chapter  1  -  INTRODUCTION 


ORGAmZATIOM 


“So  the  principles  which  are  set  forth  in  this  treatise  will,  when  taken  up  by  thoughtful  minds,  lead  to 
many  another  more  remarkable  result;  and  it  is  to  be  believed  that  it  will  be  so  on  account  of  the  nobility 
of  the  subject,  which  is  superior  to  any  other  in  nature.  ”  -Galileo  Galilei  (1638) 


1.1  COMMAND  AND  CONTROL  ASSESSMENT  CHALLENGES 

NATO  and  its  member  nations  are  in  the  midst  of  a  revolution  in  military  affairs.  There  are  three  major 
dimensions  to  this  revolution  -  a  geopolitieal  dimension,  a  technological  dimension,  and  a  closely  coupled 
conceptual  dimension.  This  multidimensional  revolution  poses  significant  new  challenges  for  analysis  in 
general  and  for  command  and  control  assessment  in  particular. 

The  changed  geopolitical  context  is  characterised  by  a  shift  from  a  preoccupation  with  a  war  involving 
NATO  and  the  Warsaw  Pact  to  a  concern  for  a  broad  range  of  smaller  military  conflicts  and  Operations 
Other  Than  War  (OOTW).  Analysts  will  increasingly  be  called  upon  to  provide  insights  into  these 
non-traditional  operations. 

Advances  in  technology,  particularly  information-related  technologies,  offer  military  organisations 
unprecedented  opportunities  to  significantly  reduce  the  fog  and  friction  traditionally  associated  with 
conflict.  At  the  same  time,  they  may  prove  to  be  challenges  in  themselves  across  a  wide  variety  of  realms 
-  technical,  organisational,  and  cultural. 

To  the  extent  that  they  can  be  achieved,  significantly  reduced  levels  of  fog  and  friction  offer  an 
opportunity  for  the  military  to  develop  new  concepts  of  operations,  new  organisational  forms,  and  new 
approaches  to  Command  and  Control  (C2),  as  well  as  to  the  processes  that  support  it.  Analysts  will  be 
increasingly  called  upon  to  work  in  this  new  conceptual  dimension  in  order  to  examine  the  impact  of  new 
information-related  capabilities  coupled  with  new  ways  of  organising  and  operating. 


1.2  DEFINITION  OF  COMMAND  AND  CONTROL 

C2  has  been  defined  by  NATO  as  Military  Function  01:  “The  Organisation,  Process,  Procedures  and 
Systems  necessary  to  allow  timely  political  and  military  decision  making  and  to  enable  military 
commanders  to  direct  and  control  military  forces”  (NATO  1996).  C2  systems  are  further  defined  in  NATO 
documents  to  include:  headquarters  facilities,  communications,  information  systems,  and  sensors  & 
warning  installations  (NATO  1998). 

Other  terms  are  used  in  NATO  member  nations  that  are  synonymous  with,  or  closely  related  to, 
C2.  These  include  Command,  Control,  and  Communications  or  Consultation  (C3),  and  Computers  (C4), 
and  Intelligence  ([C3I]  or  [C4I]),  and  Surveillance  and  Reconnaissance  (C4ISR).  The  term  CIS  is 
sometimes  used  to  refer  to  command  information  systems.  More  recently  the  term  “C2”  has  referred  to  the 
collaborative  and  consultative  processes  that  are  an  inherent  part  of  coalition  operations. 

For  the  purposes  of  this  Code  of  Best  Practice  (COBP),  the  term  C2  is  intended  to  be  an  umbrella  term 
that  encompasses  the  concepts,  issues,  organisations,  activities,  processes,  and  systems  associated  with  the 
NATO  definition  of  C2  as  well  as  the  other  terms  enumerated  above. 
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1.3  UNIQUENESS  OF  C2  ANALYSES  AND  ISSUES 

The  focus  of  military  research  and  analysis  has  predominantly  been  on  the  physical  domain.  C2  issues 
differ  in  fundamental  ways  from  physics  dominated  problems.  C2  deals  with  distributed  teams  of  humans 
operating  under  stress  and  in  a  variety  of  other  operating  conditions.  C2  problems  are  thus  dominated  by 
their  information,  behavioural,  and  cognitive  aspects  that  have  been  less  well  researched  and  understood. 
This  focus  creates  a  multidimensional,  complex  analytic  space. 

Military  operations  involve  multi-sided  dynamics  encompassing  friendly,  adversary,  and  other  actors 
including: 

•  Action-reaction  dynamics; 

•  Tightly  connected  interaction  among  subjective  elements  such  as  cultures,  morale,  doctrine, 
training,  and  experience  and  between  those  subjective  elements  and  the  combat  arena; 

•  Non-Go vemmental  Organisations  (NGO); 

•  Private  Volunteer  Organisations  (PVO); 

•  International  organisations; 

•  International  corporations;  and 

•  Trans-national,  sub-national,  criminal,  and  terrorist  organisations. 

C2  issues  are  difficult  to  decompose  and  recompose  without  committing  errors  of  logic.  Moreover, 
the  composition  rules,  by  which  the  various  factors  that  impact  C2  interact,  are  poorly  understood  except 
in  arenas  that  have  been  previously  studied  in  detail.  Finally,  the  C2  arena  is  weakly  bounded  by  issues 
that  on  initial  examination  appear  quite  finite,  but  prove  to  be  linked  to  very  high-level  factors. 
For  example,  tactical  performance  may  be  tied  to  national  culture. 

Analyses  of  C2  are  also  often  constrained  by  factors  that  are  beyond  the  boundaries  of  the  research. 
For  example,  security  policies  may  restrict  data  availability  and  otherwise  constrain  the  analysis. 
The  availability  of  data  often  limits  the  scope  of  an  analysis.  Moreover,  the  time  and  resources  available  to 
conduct  an  analysis  are  often  severely  constrained  because  the  decision  processes  being  supported  are 
being  driven  by  outside  planning,  operational,  or  budget  and  decision  processes.  This  should  be  seen  as  a 
challenge  rather  than  a  problem.  Uncertainty  and  risk  associated  with  a  lack  of  appropriate  data  need  to  be 
embraced  as  part  of  the  analytical  approach.  It  is  unreasonable  to  expect  that  data  would  be  available  for 
the  performance  of  future  systems  and  processes  that  do  not  yet  exist.  An  experimental  component  and  a 
modelling  and  simulation  component  need  to  be  integrated  into  modem  C2  analyses  in  order  to  close  the 
gap  in  knowledge  and  data. 

Finally,  because  of  the  complexity  of  C2  processes  and  systems,  analysis  in  this  area  requires  the  ability  to 
understand  how  Dimensional  Parameters  (DP),  Measures  of  Performance  (MoP),  Measures  of  C2 
Effectiveness  (MoCE),  Measures  of  Force  Effectiveness  (MoFE),  and  Measures  of  Policy  Effectiveness 
(MoPE)  are  linked  and  impact  on  one  another.  The  cumulative  set  of  these  measures  is  denoted  as 
Measures  of  Merit  (MoM)  in  the  COBP.  Determining  the  precise  nature  of  these  relationships  nearly 
always  proves  to  be  an  analytic  challenge. 

Taken  together,  all  these  factors  mean  that  C2  modelling  and  analysis  are  more  uncertain  and  therefore 
more  prone  to  risk  than  their  equivalents  in  conventional  weapon  and  platform  analyses.  Indeed,  C2  issues 
have  long  been  regarded  as  difficult  to  analyse.  Many  operational  analysis  (OA)  studies  have  simply 
assumed  perfect  C2  in  order  to  focus  on  other  variables.  As  a  result  of  these  characteristics  of  C2  analysis, 
these  endeavours  will  require  a  heavy  element  of  research  within  each  analysis.  This  COBP  is  intended  to 
assist  the  community  in  dealing  with,  and  overcoming,  the  barriers  to  effective  analysis  of  C2. 
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1.4  DIFFERENCES  ACROSS  THE  MISSION  SPECTRUM 

There  are  significant  differences  among  the  different  parts  of  the  mission  spectrum  (e.g.  MoM)  that  the 
assessment  team  needs  to  take  into  consideration.  Table  1.1  highlights  the  differences  between  traditional 
combat  and  OOTW. 

Table  1.1:  Comparison  of  Symmetric,  Conventional 
Warfare  and  OOTW  Missions  &  Principles  Mission 


Factor  Symmetric,  Conventional  OOTW 


Mission/Operation 

Stability 

Relatively  stable 

May  be  more  dynamic 

Focus 

Enemy 

No  traditional  opponent 

Commitment 

Common  (military) 

Uncertain  (political/military) 

Principles 

Unity 

Of  command 

Of  purpose 

Decisionmaking 

Hierarchical 

Consensus 

Operations 

Surprise,  secrecy 

Transparency 

Information 

Nature  of  the  problem 

Known  unknowns 

Unknown  unknowns 

Key  question 

How  to  get  information 

What  information  to  get 

Focus 

Enemy  military 

Military/political/economic/ 
social  factors 

Situation  awareness 

Common  air-land-sea 

Limited  dissemination, 
more  complex 

Databases 

Very  large,  well  structured 

Larger,  less  structured 

Analysis 

Unit 

Battalion  level  entity 

More  behavioural 

Ease  in  integration 

Relatively  easy 

Very  difficult 

Focus 

Military  (systems,  organisations) 

Political/Military  and  societal 

Approach 

Traditional  operation  analysis 

“Softer”  analysis 

In  symmetric  conventional  warfare,  the  mission  tends  to  be  relatively  stable,  there  is  a  clear  focus  on  the 
enemy,  and  the  military  has  a  common  understanding  and  commitment'.  Conversely,  in  OOTW  the 
mission  is  often  more  dynamic.  This  is  captured  by  the  often  pejorative  term  “mission  creep.”  In  many  of 
the  operations  in  question  there  is  no  “enemy.”  This  is  obviously  true  for  operations  such  as  humanitarian 
assistance  and  disaster  relief  In  addition,  peacekeeping  activities  involve  protagonists  who  must  be  treated 
even-handedly  if  the  operation  is  to  be  successful.  In  the  latter  missions,  political-military  ambiguities 
frequently  result  in  uncertain  understanding  of  the  goals  and  objectives  of  the  mission  and  a  limited 
commitment  (Starr,  Haut  &  Hughes,  1997). 


'  As  an  illustration.  General  Colin  Powell,  then  Chairman,  Joint  Chiefs  of  Staff,  summarised  the  mission  in  Desert  Storm  by 
stating  that  “First,  we  will  cut  off  the  enemy  and  then  we  will  kill  it.”  (Pentagon  Briefing,  Wednesday  January  23,  1991.) 

2 

As  an  example,  the  US  Congress  has  continually  sought  to  impose  an  arbitrary  deadline  for  US  forces  to  withdraw  from 
Bosnia. 
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1.4.1  Principles 

Military  theorists  have  frequently  propounded  basic  principles  of  conventional  warfare.  Three  often  cited 
principles  include  the  need  for  unity  of  command,  the  importance  of  hierarchical  decisionmaking, 
and  the  criticality  of  achieving  surprise  in  operations.  A  recent  book  has  proposed  alternative  principles 
for  OOTW  (Alberts  &  Hayes,  1995).  It  cites  the  need  for  unity  of  purpose,  consensus  decisionmaking, 
and  transparency  of  operations. 

1.4.2  Information 

In  conventional  warfare,  the  issue  of  information  gathering  and  management  focuses  on  the  issue  of 
“known  unknowns”  (e.g.  Where  are  the  enemy’s  battalions?).  For  that  case,  the  key  question  is  how  to  get 
the  needed  information  (e.g.  What  are  the  key  signatures  for  the  targets  in  question?  What  sensors  should 
we  task  to  exploit  those  signatures?).  Clearly,  the  focus  is  on  the  enemy  military  and  one  objective  is  to 
assemble  a  complete,  timely,  and  accurate  common  picture  of  the  air-land-sea  situation.  The  result  is  a 
very  large,  time-sensitive  database,  but  one  that  is  relatively  well  structured  (e.g.  enemy  order  of  battle). 
Conversely,  in  OOTW,  the  problem  of  information  gathering  and  management  is  dominated  by  “unknown 
unknowns.”  Thus,  the  primary  question  to  address  is  what  information  to  get.  The  information  focus  is 
much  more  diffuse  because  of  the  myriad  of  military,  political,  economic,  and  social  factors  that  must  be 
considered.  Consequently,  situation  awareness  is  much  more  complex.  Political  considerations  often  make 
it  prudent  to  limit  the  dissemination  of  information,  creating  a  tension  between  the  desire  to  create  shared 
awareness  by  increasing  information  sharing  and  the  need,  for  political  and/or  security  reasons,  to  limit 
information  sharing.  The  resulting  databases  are  frequently  larger  and  less  structured. 

1.4.3  Analysis 

Over  many  years,  the  military  operations  research  community  has  become  relatively  adept  at  analysing 
key  aspects  of  symmetric  conventional  warfare.  As  an  illustration,  analyses  of  ground  warfare  often  focus 
on  battalion-level  operations  and  techniques  have  emerged  to  integrate  across  those  results  to  derive 
insights  into  campaign  outcomes.  The  focus  is  on  military  systems  and  organisations,  and  the  techniques 
in  question  involve  a  broad  set  of  methods  (e.g.  mathematical  programming,  decision  theoretic 
approaches)  and  tools  (e.g.  models  and  simulations).  Analyses  of  C2  issues  remain  among  the  most 
challenging,  even  in  warfare  contexts.  In  addition,  analyses  of  OOTW  often  require  consideration  of 
individual  behaviour.  It  has  proven  very  difficult  to  integrate  across  these  results  to  derive  a 
comprehensive  understanding  of  the  problem.  The  issue  is  compounded  by  the  many  factors  that  have  to 
be  considered  in  the  analysis  process  (e.g.  military,  political,  economic,  social).  This  has  led  to  the 
application  of  “softer”  analytic  approaches  (e.g.  extensive  reliance  on  expert  elicitation).  Moreover, 
the  very  nature  of  warfare  appears  to  be  changing.  For  example,  asymmetric  threats  are  becoming  more 
common,  information  technologies  are  impacting  C2  processes,  and  organic  structure  and  dynamics  are 
changing  rapidly  and  in  ways  we  do  not  fully  understand. 


1,5  TYPES  OF  C2  ASSESSMENTS 

The  assessment  team  could  be  called  upon  to  support  a  wide  variety  of  sponsors  (e.g.  acquirers  of  C2 
systems,  long  range  planners  and  programmers,  developers  of  requirements,  operational  commanders, 
and  trainers).  These  sponsors  will  bring  different  problems  to  the  assessment  team  (e.g.  assessment  of 
alternative  systems  or  concepts,  identification  and  selection  of  alternative  courses  of  action  in  an 
operational  context).  Some  of  these  will  deal  with  a  specific  mission  (e.g.  air  defence)  while  others  will 
need  to  deal  with  the  entire  mission  spectrum  from  forward  presence  to  high  intensity  conventional  war. 


1  -4 


RTO-TR-081 


INTRODUCTION 


Specific  problems  that  the  team  may  be  called  upon  to  address: 

Requirements  Analysis 

•  Derivation  of  specific  C2  requirements  from  broad  statements  of  objectives;  and 

•  The  establishment  of  a  minimum,  standard,  or  expected  level  of  performance. 

Assessment  of  Alternatives 

•  Comparison  and  selection  of  alternative  systems  that  may  be  very  dissimilar  but  are  designed  to 
achieve  a  similar  purpose; 

•  Assessment  of  the  utilisation  of  a  system  in  a  new  or  unexpected  application  domain  or  mission; 

•  Trade-offs  between  C2  systems  and  combat  systems; 

•  Analysis  of  the  impact  of  an  organisational  change; 

•  Determination  of  the  most  cost-effective  approach  to  achieving  the  desired  objective;  and 

•  Comparison  of  a  replacement  system  or  components  of  a  system. 

Research  Issues 

•  Effectiveness  of  human  decisionmaking  as  a  function  of  system  performance  or  other  factors; 

•  Effectiveness  of  C2  training;  and 

•  Impact  of  collaboration  on  C2  quality. 

Support  to  Operations 

•  Course  of  action  analysis; 

•  Real  time  assessment  of  mission  effectiveness;  and 

•  Rehearsal  assessment. 


1.6  PURPOSE  AND  SCOPE  OF  THE  COBP 

This  COBP  offers  broad  guidance  on  the  assessment  of  C2  for  the  purposes  of  supporting  a  wide  variety 
of  decisionmakers  and  the  conduct  of  C2  research  described  above.  It  should  be  noted  that  this  COBP  is 
focused  upon  the  assessment  challenges  associated  with  the  nature  of  C2  and  does  not  attempt  to 
specifically  address  the  unique  properties  and  constraints  associated  with  each  of  the  many  C2-related 
problem  domains. 

Given  the  increasing  interdependence  among  the  elements  of  a  mission  capability  package^  (organisation; 
doctrine;  C2  concepts,  processes,  systems;  materiel;  education;  training;  and  forces),  C2-related  analysis 
cannot  easily  be  done  in  isolation  from  a  more  comprehensive  mission  analysis.  This  COBP  is  meant  to 
support  analyses  that  go  beyond  the  traditional  boundaries  of  C2  analyses. 

This  new  version  of  the  COBP  for  C2  assessment  was  developed  by  SAS-026  building  upon  the  initial 
version  of  the  COBP  produced  by  SAS-002.  This  new  COBP  is  a  synthesis  of  decades  of  expertise  from 
various  countries  and  hundreds  of  analyses.  The  COBP  was  developed  using  a  set  of  case  studies  to  test 
out  the  varied  advice  and  guidance  received,  and  incorporates  feedback  from  users  of  the  initial  version. 
Easily,  SAS-039  provided  a  peer  review  of  the  final  draft  product. 


^  Mission  capability  packages  include  all  of  the  elements  necessary  for  an  operation  (Alberts,  1996). 
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The  earlier  version  focused  on  the  analysis  of  ground  forces  at  a  tactical  echelon  in  mid  to  high  intensity 
conflicts.  Consequently,  the  initial  version  of  the  COBP  did  not  completely  address  the  full  range  of 
important  issues  related  to  C2.  In  developing  this  new  version  of  the  COBP,  SAS-026  explicitly  focused 
upon  OOTW,  the  impact  of  significantly  improved  information  related  capabilities,  and  their  implications 
for  military  organisations  and  operations.  In  addition,  SAS-026  was  cognisant  of  the  fact  that  NATO 
operations  are  likely  to  include  coalitions  of  the  willing,  which  might  involve  Partnership  for  Peace  (PfP) 
nations,  others  partners  outside  of  NATO,  international  organisations,  and  NGOs.  NATO  operations  may 
also  be  “out  of  area.” 

Feedback  from  users  of  the  original  COBP  also  identified  a  number  of  ways  in  which  the  original  COBP 
could  be  improved.  These  areas  were  addressed  during  the  development  of  this  version  of  the  COBP. 

Cost  analyses  continue  to  be  explicitly  excluded  for  two  reasons.  First,  cost  analysis  is  a  mature  discipline 
that  experienced  operational  analysts  already  practice.  Hence,  C2  issues  are  not  unique  in  the  arena. 
Second,  most  nations  have  already  developed  approaches  to  cost  analysis  and  cost  effectiveness  that  are 
consistent  with  their  national  approaches  to  accounting.  Because  these  national  practices  differ  among 
NATO  members,  no  single  approach  would  be  appropriate. 

As  this  COBP  is  being  drafted,  novel  experiments  with  new  information-related  capabilities,  particularly 
networking  and  ways  to  accomplish  their  assigned  tasks  abound.  Advances  in  technology  are  expected  to 
continue  at  an  increasing  rate  and  spur  both  sustaining  and  disruptive  innovation  in  military  organisations. 
It  is  to  be  expected  that  this  COBP  will  need  to  be  periodically  revisited  in  light  of  these  developments. 


1.7  OVERVIEW  OF  COBP  ASSESSMENT  PHILOSOPHY 

The  COBP  assumes  that  the  objective  of  a  C2  system  is  to  exercise  control  over  its  environment,  through 
either  adaptive  or  reactive  control  mechanisms,  or  some  combination  of  those  two  approaches. 
This  provides  the  context  and  point  of  departure  for  the  assessment  of  C2. 

Analysis  of  C2  should  consider  all  the  relevant  actors,  military  command  levels,  and  functions  involved 
and  should  investigate  issues  of  integration  across  disparate  organisations,  military  command  levels,  and 
functional  domains  over  time.  Consideration  should  also  be  given  to  the  robustness  and  security  of 
information  systems  and  to  human  computer  interface  issues.  Human  behavioural,  physiological, 
and  cognitive  factors,  along  with  organisational  and  doctrinal  issues,  must  be  considered  in  C2  analyses. 

C2  assessments  must  also  consider  a  range  of  missions,  adversary  capabilities,  and  adversary  behaviours. 
Moreover,  it  must  be  understood  that  adversaries  will  use  asymmetric  tactics  and  techniques  to  deny  or 
exploit  differences  in  technology,  force  size,  information  systems,  or  cultural  factors.  Hence,  scenarios  and 
analyses  that  deal  with  an  appropriate  set  of  all  these  dimensions  should  be  considered  in  either  the  main 
research  design  or  in  the  excursions  to  assess  risks  and  uncertainty. 

The  evaluation  of  C2  issues  depends  in  important  ways  on  both  distinguishing  and  linking  dimensional 
parameters,  measures  of  performance,  measures  of  C2  effectiveness,  and  measures  of  force  and  policy 
effectiveness.  Modelling  and  other  tools  must  be  designed  to  support  this  requirement. 

Tools  and  data  used  in  C2  analysis  should  conform  to  good  OA  processes  and  practices  and,  to  the  extent 
feasible,  should  be  subject  to  Model  Verification,  Validation,  and  Accreditation  (VV&A)  and  to  Data 
Verification,  Validation,  and  Certification  (VV&C). 

Interoperable  analytical  infrastructures  (e.g.  data  dictionaries,  glossaries,  models,  tools,  data  sets) 
are  necessary  to  facilitate  the  efficient  proliferation  and  reuse  of  study  results  and  data  within  the  broader 
interdisciplinary  research  community. 
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Because  the  complexity  of  C2  and  the  requirements  for  its  analysis  are  often  underestimated  by 
decisionmakers,  a  continuing  dialogue  between  analysts  and  those  decisionmakers  is  necessary  both  to 
scope  the  problem  properly  and  to  ensure  that  the  analytic  results  are  properly  understood.  Part  of  this 
process  includes  performing  sensitivity  analyses  and  other  common  practices  designed  to  ensure  the 
validity  and  reliability  of  the  results. 

Changes  to  C2  systems  will  often  lead  to  changes  in  military  concepts,  command  approaches,  doctrine. 
Tactics,  Techniques,  and  Procedures  (TTP),  and  related  factors,  which  must  also  be  considered  in  the 
analysis. 


1.8  CURRENT  STATE  OF  PRACTICE  IN  C2  ANALYSIS 

Assessment  of  C2  issues  typically  employs  classic  tools  of  OA.  Relatively  few  specialised  tools  and 
methods  have  been  developed  for  C2.  Moreover,  those  specialised  tools  generated  to  deal  with  the  unique 
aspects  of  C2-focused  research  are  generally  not  as  well  understood  as  those  used  in  more  traditional 
warfare  modelling  domains.  C2  analysts  will  often  find  themselves  having  to  develop  tools  and 
approaches  appropriate  for  their  research  agendas.  However,  a  general  analytic  process  can  be  identified 
that  will  enhance  the  likelihood  that  an  OA  analyst  can  conduct  successful  analyses. 


1.9  ORGANISATION  OF  THE  COBP 

This  COBP  is  organised  into  four  themes.  The  first  theme  deals  with  study  dynamics,  problem 
formulation,  and  the  development  of  a  solution  strategy.  The  second  theme  identifies  and  discusses  in 
depth  the  essential  elements  of  assessment:  measures  of  merit,  scenarios,  human  and  organisational  issues, 
data,  and  tools.  The  third  theme  addresses  issues  related  to  risk  and  uncertainty  while  the  final  theme 
describes  the  range  of  assessment  products. 

This  represents  a  significant  enhancement  of  the  initial  COBP.  In  particular  the  first,  third  and  fourth 
themes  were  not  treated  in  detail  in  the  initial  version  of  the  COPB.  In  addition,  material  has  been  added  to 
the  second  theme  to  address  the  unique  assessment  challenges  associated  with  OOTW. 


1.10  BRIEF  HISTORY  OF  SAS-026 

SAS-026  builds  upon  almost  a  decade  of  work  that  began  with  the  formation  of  the  Ad  Hoc  Working 
Group  on  the  Impact  of  C3I  on  the  Battlefield  by  Panel  7  of  the  NATO  Defence  Research  Group  in  1991 
to  assess  the  state  of  the  art  in  C2  analysis.  Based  on  the  recommendations  of  the  Ad  Hoc  Working  Group, 
Panel  7  constituted  Research  Study  Group- 19  (RSG-19)  to  address  issues  of  methodology,  measures  of 
merit,  and  tools  and  analysis.  The  panel  also  addressed  issues  of  improving  a  nation’s  capability  to 
examine  C2  acquisition  and  decisionmaking.  At  the  October  1995  RSG-19  planning  meeting,  the  group 
determined  that  the  primary  product  of  RSG-19  was  to  be  a  Code  of  Best  Practice  for  assessing  C2. 
As  part  of  selected  RSG-19  meetings,  workshops  would  be  conducted  to  support  the  development 
of  the  major  sections  of  the  COBP.  Workshops  were  conducted  on  Measures  of  Merit  (Canada), 
Scenario  Development  (Netherlands),  C3I  Systems,  Structures,  Organisations,  and  Staff  Performance 
Evaluations  (Norway),  and  Models  Used  for  C3  Systems  and  Analysis  (US/UK).  Representatives  from  the 
nations  in  parentheses  took  the  lead  in  organising  the  workshops  and  summarising  the  results.  The  minutes 
of  the  workshops  provide  further  illustrations  of  the  techniques  presented  in  the  COBP. 

At  the  October  1996  meeting,  the  group  took  up  a  request  by  Panel  7  to  conduct  a  symposium  on 
modelling  and  analysis  of  C3I,  which  was  scheduled  at  the  July  1997  meeting  for  January  1999. 
This  symposium  was  a  forum  for  presentation  and  discussion  of  the  COBP  and  related  topics. 
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At  the  July  1997  meeting,  in  response  to  a  query  by  Panel  7,  the  group  discussed,  acknowledged, 
and  agreed  on  the  need  for  a  follow-on  group  to  SAS-002.  An  exploratory  group  on  organisational  change 
(SAS-E05)  was  formed  to  recommend  a  way  ahead. 

SAS-E05  recommended  the  formation  for  a  follow-on  activity  to  SAS-002  to  accomplish  four  objectives: 

•  Demonstrate  and  assess  the  initial  version  of  the  COBP; 

•  Revise  and  extend  the  COBP; 

•  Identity  research  areas;  and 

•  Facilitate  the  adoption  of  the  COBP. 

The  SAS  panel  concurred  in  May  1999  and  approved  the  formation  of  SAS-026,  which  began  its 
2  1/2 -year  plan  of  work  in  a  symposium  in  January  2000. 


1.11  CHAPTER  1  ACRONYMS 


C2 

C3(I) 

C4(I) 

C4ISR 

CIS 

COBP 

DP 

MoCE 

MoFE 

MoM 

MoP 

MoPE 

NGO 

OA 

OOTW 

PfP 

PVO 

RSG-I9 

TTP 

VV&A 

vv&c 


Command  and  Control 

Command,  Control,  Communications  (and  Intelligence) 

Command,  Control,  Communications,  and  Computers  (and  Intelligence) 

Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance  and 
Reconnaissance 

Command  Information  Systems 

Code  of  Best  Practice 

Dimensional  Parameters 

Measures  of  C2  Effectiveness 

Measures  of  Force  Effectiveness 

Measures  of  Merit 

Measures  of  Performance 

Measures  of  Policy  Effectiveness 

Non-Govemmental  Organisations 

Operational  Analysis 

Operations  Other  Than  War 

Partnership  for  Peace 

Private  Volunteer  Organisations 

Research  Study  Group- 1 9 

Tactics,  Techniques,  and  Procedures 

Verification,  Validation,  and  Accreditation 

Verification,  Validation,  and  Certification 
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Chapter  2  -  PREPARING  FOR  SUCCESS:  ASSESSMENT 
PARTICIPANTS,  RELATIONSHIPS,  AND  DYNAMICS 


‘‘For  hypotheses  ought ...  to  explain  the  properties  of  things  and  not  attempt  to  predetermine  them  except 
in  so  far  as  they  can  be  an  aid  to  experiments.  ”  -Isaac  Newton  (1687) 

“We  have  run  out  of  Money  -  Now  we  have  to  think.  ”  -Ernest  Rutherford  (1871-1937) 


2.1  OVERVIEW 

This  chapter  is  organised  into  three  parts.  The  first  discusses  the  roles  played  by  the  significant  players 
associated  with  a  Command  and  Control  (C2)  assessment  and  how  these  roles  affect  the  design  and 
conduct  of  the  assessment.  The  second  part  identifies  the  major  phases  of  a  C2  assessment  and  their 
iterative  nature.  The  concluding  section  addresses  the  subject  of  professional  ethics  and  standards  of 
conduct. 


2.2  ASSESSMENT  PARTICIPANTS 

Like  their  subject,  the  organisation  of  C2  studies  involves  complex  interrelationships.  It  is  crucial  for  the 
analytical  team  to  establish  which  individuals  and  organisations  are  involved  at  an  early  stage  of  the  study. 
It  is  prudent  for  the  analytical  team  to  map  the  roles  described  below  onto  the  individuals  and 
organisations  involved  and  to  understand  their  interrelationships.  An  example  of  such  a  mapping  is  at 
Figure  2.1.  Appendix  1  to  this  chapter  provides  a  brief  explanation  of  the  organisations  involved. 

Due  to  the  dynamic  nature  of  such  projects,  those  involved  should  not  be  surprised  if  the  nature  of  the 
teams  involved  might  have  to  expand  or  change  with  time. 

2.2.1  Assessment  Team 

The  assessment  team  is  working  for  the  sponsor  or  client  (sponsor).  The  team  consists  of  a  senior  team 
leader  (who  may  also  be  referred  to  as  the  project  manager),  a  core  set  of  analysts,  subject  matter  experts 
including  military  officers,  and  supporting  staff  who  are  working  on  the  study  on  a  day  to  day  basis. 
The  team  provide  the  legitimacy  and  authority  for  the  study.  The  sponsor  will  provide  the  terms  of 
reference,  access  to  needed  information,  and  identify  the  desired  products.  It  is  important  for  the  analytical 
team  to  understand  exactly  why  the  sponsor  wants  the  study  and  what  the  sponsor  wants  to  do  with  the 
results. 

2.2.2  Decisionmakers  or  Problem  Owners 

The  decisionmakers  are  the  individuals  or  organisations  that  are  expected  to  make  decisions  wholly  or 
partially  based  on  the  output  or  findings  of  the  study.  If  there  is  no  decision  to  be  made  (i.e.  this  is  an 
exploratory  study)  then  the  decisionmakers  could  be  referred  to  as  problem-owners.  It  is  important  for  the 
assessment  team  to  understand  exactly  what  type  of  assessment  the  decisionmakers  want  the  study  to 
support.  The  decisionmakers  may  or  may  not  be  in  command  of  or  part  of  the  sponsor’s  organisation. 
Complex  problems  may  arise  when  the  decisionmaker  is  several  steps  laterally  away  in  the  organisation 
from  the  sponsor  and  study  team. 
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2.2.3  Stakeholders 

The  stakeholders  are  the  persons  or  organisations  that  are  directly  or  indirectly  affected  by  the  study 
outcome.  Stakeholders  may  also  play  other  roles.  The  assessment  team  must  be  aware  of  the  potential  for 
conflict  when  the  stakeholders  do  not  include  the  sponsor  or  decisionmaker.  Complex  problems  may  arise 
in  the  provision  of  data  for  the  study,  as  it  is  the  stakeholders  who  may  have  to  provide  the  data,  set  the 
security  or  releasibility  of  that  data  (and  hence  the  study),  and/or  agree  that  the  data  are  representative. 
For  these  reasons  it  is  essential  that  the  analysts  establish  a  working  relationship  with  the  stakeholders 
early  in  the  process. 

2.2.4  Bill  Payer 

The  bill  payer  is  the  organisation  or  individual  official  paying  for  the  study.  It  is  important  for  the 
assessment  team  to  know  the  level  of  resources  available.  Bill  payers  will  normally  have  a  direct  interest 
in  the  outcome  of  the  study  and  may  be  one  or  more  of  the  other  players.  Contractual  authorities  have  the 
legal  authority  to  let  contracts  on  behalf  of  the  bill  payer. 

2.2.5  Existing  Study  Teams 

The  assessment  team  must  be  aware  of  and  sensitive  to  the  existence  of  teams  in  other  related  study  areas. 
Should  such  teams  exist,  the  assessment  team  should  endeavour  to  exploit  the  work  done  and  available 
skills  and  techniques.  Such  external  teams  may  also  be  appropriate  for  membership  in  peer  reviews. 

2.2.6  Future  Study  Teams 

The  assessment  team  must  be  aware  of  and  sensitive  to  the  needs  of  future  analyses  and  assessments. 
Data  collection,  method  documentation,  and  the  archiving  of  data,  methods,  models  and  results 
are  fundamental  responsibilities  of  all  professional  analysts.  Method  and  data  should  be  (as  far  as  is 
practicable)  disseminated  and  published. 

2.2.7  Peer  Reviewers 

Outside  experts  brought  in  to  look  at  the  work  and  provide  constructive  criticisms  are  called  peer 
reviewers.  Peer  review  teams  could  be  composed  of  specialists  and  other  study  teams  in  related  subject 
areas  and  should  include  representatives  from  all  key  disciplines  in  the  assessment. 

2.2.8  Data  Providers 

Data  providers  are  the  individuals  and  organisations  that  possess  data  and  information  useful  to  the 
assessment  team.  Many  of  these  will  be  stakeholders.  The  motivation  to  provide  data  to  the  study  must  be 
developed  by  the  analytical  team  and  the  sponsor. 

2.2.9  Assumptiou  Providers 

Assumption  providers  are  the  individuals  or  organisations  that  can  provide  “givens”  such  as  future 
doctrine,  performance  data,  force  mixes,  organisational  structures,  and  scenarios.  Creation  of  a  positive 
relationship  with  these  organisations  is  important  to  the  study. 

2.2.10  Data  Collectors 

In  some  C2  analyses,  where  data  must  be  extracted  from  real  world  experiences,  exercises,  experiments, 
and  wargames,  teams  of  data  collectors  and  subject  matter  experts  will  be  required.  The  identification  of 
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people  with  the  appropriate  background  and  training  as  data  collectors  are  important  elements  of  such 
studies. 


2.3  RELATIONSHIPS  AMONG  PARTICIPANTS  AND  THE  CONDUCT  OF 
THE  ASSESSMENT 

2,3,1  Relationships 

Figure  2.1  below  illustrates  how  complicated  the  participant  roles  and  relationships  can  be  in  a  real 
C2  assessment.  This  particular  figure  represents  the  organisations  involved  in  the  recent  Immediate 
Reaction  Task  Force  (Land)  (IRTF(L))  C2  Concept  evaluation  that  completed  at  the  end  of  2001 
(Candan  &  Lambert,  2002).  Appendix  1  to  this  chapter  provides  a  brief  explanation  of  the  organisations 
involved.  Although  not  all  projects  will  be  this  complex,  many  important  C2  assessments  will. 


Figure  2.1 :  Example  of  the  Mapping  of  Roles  onto  Players  -  for  a  Complex  C2  Project 
(the  Evaluation  of  Immediate  Reaction  Task  Force  (Land)  C2  Concept). 


Through  the  prudent  act  of  mapping  the  roles  of  the  participants  of  the  study,  the  potential  conflicts  of 
interest  and  complex  interactions  can  be  identified.  One  method  to  mitigate  these  is  to  present  or  conduct 
this  activity  openly  and  discuss  with  all  involved  so  that  all  potentially  affected  participants  are  aware  of 
the  possibility  of  future  conflict  and  the  fact  that  all  participants  fall  into  one  or  more  roles  within  a 
project. 

In  the  event  of  conflict  with  other  participants  in  the  project  the  assessment  team  address  the  issues  in  a 
neutral  and  independent  manner. 

2,3,2  Understanding  the  Context  of  the  Assessment 

The  relationship  among  the  assessment  team,  the  key  sponsor,  and  the  stakeholders  is  of  paramount 
importance  and  perhaps,  more  than  any  other  single  factor,  will  influence  the  course  and  success  of  the 
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effort.  Accordingly,  adequate  attention  needs  to  be  paid  to  understanding  the  situation  facing  the  key 
sponsor  and  stakeholders  as  much  as  the  subject  under  study. 

The  assessment  team  should  be  aware  that  the  different  participants  will  have  divergent 
perspectives  and  may  have  divergent  agendas. 

Therefore  the  assessment  team  should  understand  the  background  of  the  individuals  involved, 
their  organisational  settings,  roles  and  responsibilities,  their  history,  and  their  current  situation.  Contact 
with  analysts  who  have  worked  with  this  sponsor  and  review  of  prior  analyses  for  this  sponsor  facilitate 
this  objective. 

It  is  good  practice  to  build  and  maintain  long-term  relationships  with  the  sponsor  and  stakeholder 
organisations.  This  will  yield  substantial  dividends  in  the  form  of  easier  communication,  greater 
trust,  and  stronger  support. 

2.3.3  A  Continuing  Dialogue 

It  is  important  that  a  dialogue  with  the  sponsor  and  stakeholders  is  maintained  by  the  assessment  team 
throughout  the  study.  As  there  is  no  single  “language”  that  will  describe  the  study  problem,  it  is  important 
to  spend  time  at  the  beginning  of  a  study  to  establish  a  common  “language”  that  both  the  assessment  team 
and  the  sponsor  and  stakeholder  can  understand.  This  point  may  seem  obvious  in  a  NATO  setting  in  which 
the  participants  speak  many  different  natural  languages.  However,  it  is  equally  important  in  a  single 
language  setting  because  common  words  and  phrases  have  different  meanings  for  different  organisations, 
services,  and  even  individuals  within  a  single  organisation.  Regular  meetings  and  contact  will  minimise 
misunderstandings.  From  a  professional  point  of  view.  Operations  Research  (OR)  and  Operations  Analysis 
(OA)  analysts  will  always  wish  to  inform  the  sponsor  and  stakeholders  of  key  developments  and/or 
challenges  as  the  study  unfolds.  Regular  and  routine  interactions  need  to  be  built  into  the  project  plan. 
If  there  are  multiple  sponsors  and  stakeholders  and  other  key  actors,  the  assessment  team  should  try  to 
meet  them  jointly,  particularly  when  decisions  need  to  be  made.  Separate  meetings  will  often  lead  to 
inconsistent  guidance  and  will  place  the  assessment  team  in  a  position  of  trying  to  accommodate  differing 
interests. 

The  development  of  a  collectively  agreed  upon  Study  Glossary  that  captures  the  definitions  of 
words,  phrases  and  acronyms  used  in  the  study  is  a  useful  tool. 

2.3.4  Terms  of  Reference 

A  good  term  of  reference  covers  goals,  scope,  products,  schedule,  and  resources.  These  will  determine  the 
focus  of  the  assessment  and  establish  limits  or  freedoms  granted  to  the  assessment  team  within  the 
sponsor’s  and  stakeholder’s  organisations.  Letters  of  introduction  and  instructions  to  actors  within  the 
sponsor  and  stakeholder  organisations  may  also  be  useful. 

2.3.5  Understanding  How  the  Output  of  the  Study  Will  he  Used 

It  is  important  to  know  at  an  early  stage  in  the  project  what  the  products  of  the  study  are  to  be  used  for  by 
the  sponsor  and  stakeholder  organisations.  The  expected  end  product  will  set  the  tone  and  relative 
importance  of  the  project  in  the  eyes  of  the  sponsor,  stakeholder,  and  other  actors.  The  assessment  team 
needs  to  establish  and  understand  the  products  needed  or  desired  by  the  sponsor  and  stakeholder. 
For  example,  a  study  could  be  used  to  affect  a  significant  impact  on  the  stakeholder’s  domain,  gain  a 
greater  understanding  of  the  issues,  produce  an  improved  capability  to  perform  future  work,  and/or  make 
contributions  to  the  body  of  knowledge. 
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2.3.6  Budget 

The  sponsor  will  have  limited  resourees  with  a  study  budget  in  mind.  When  the  sponsor’s  resources  are 
limited  to  a  level  below  what  is  required  to  support  a  quality  study,  the  assessment  team  will  need  to 
suggest  strategies  to  address  the  shortfall.  Alternative  approaches  include  decomposing  the  problem  and 
only  undertaking  the  core  part  of  the  study  that  is  affordable,  linking  the  sponsor  to  other  actors  that  have 
an  interest  in  the  same  or  a  similar  problem  and  who  could  contribute  resources,  and/or  stretching  the 
project  over  a  longer  time  so  that  resources  from  future  budget  cycles  become  available.  In  developing 
strategies  that  involve  doing  only  a  part  of  the  study  to  satisfy  budget  constraints,  care  must  be  taken  to 
ensure  that  the  product  that  will  be  produced  provides  a  meaningful  answer  or  contribution  and  does  not 
depend  upon  a  follow-on  effort  that  may  or  may  not  be  funded. 

It  may  take  a  complete  iteration  of  the  assessment  phases  of  the  project  to  establish  the  complete  scope  of 
the  project  and  the  resources  required.  Therefore  it  is  good  practice  in  large  C2  projects  to  allow  the 
assessment  team  to  perform  a  rapid  first  pass  of  all  the  phases  of  the  project  to  help  establish  the  budget 
required.  This  is  contrary  to  the  usual  practice  of  setting  the  budget  in  stone  immediately  following  the 
initial  Problem  Formulation  or  Solution  Strategy  phase  (see  Section  2-E  below). 

2.3.7  Relationships 

Figure  2. 1  below  illustrates  how  complicated  the  participant  roles  and  relationships  can  be  in  a  practical 
C2  assessment.  This  particular  figure  represents  the  organisations  involved  in  the  recent  Immediate 
Reaction  Task  Force  (Land)  (IRTF(L))  C2  Concept  evaluation  that  completed  at  the  end  of  2001 
(Candan  &  Lambert,  2002).  Although  not  all  projects  will  be  this  complex,  many  important  C2 
assessments  will. 


2,4  BUILDING  AN  ASSESSMENT  TEAM 
2,4,1  Skills  Available  to  the  Assessment  Team 

Following  initial  problem  formulation  (Chapter  3),  the  precise  skills  and  experience  required  by  the 
assessment  team  will  need  to  be  established.  Typically,  the  assessment  team  will  need  to  be 
interdisciplinary.  The  wide  range  of  skills  and  experience  required  can  be  allocated  between  a  full  time 
core  team  and  a  collection  of  consultants  or  part-time  team  members.  As  an  example,  an  ideal  breakdown 
of  the  skills  available  to  the  assessment  team  involved  in  the  evaluation  of  the  Immediate  Reaction  Task 
Force  (Land)  C2  concept  study'  is  given  below: 

Skills:  Core  Team 

•  Project  management; 

•  OR/OA  skills:  simulation,  wargaming,  mathematical  programming,  database  creation  and 
management,  brainstorming  and  problem  structuring,  scientific/military  report  writing/editing; 

•  Cross  military  experience  -  i.e.,  OR/OA  personnel  with  military  experience  or  military  personnel 
with  OR/OA  knowledge; 

•  Organisational  theory;  and 

•  Data  collection  (e.g.  questionnaire  and  form  design). 


'  As  developed  independently  by  the  SAS-026  panel  in  February  2001  in  response  to  a  presentation  in  the  IRTF(L)  project. 
In  fact,  this  was  fairly  close  to  what  was  available  to  the  team. 
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Skills  and  Experience:  Consultants  and  Part-time  Team  Members 

•  Military  (or  access  to  practical  experience  of  problem  under  study); 

•  Training  and  exercise  planning  (if  an  exercise  is  to  be  used  as  a  vehicle  for  the  study); 

•  Communications  and  information  systems  specialists  for  the  systems  of  the  organisation  under 
study; 

•  Human  computer  interface  expertise; 

•  Operations  Other  Than  War  (OOTW)  related  issues  (e.g.  C2/Headquarters  [HQ],  media, 
civil-military  cooperation  -  theory  and  practice); 

•  Social  scientists  (e.g.  political,  psychological,  economic,  cultural,  legal); 

•  Military  history; 

•  Command  experience; 

•  Deployment  analysis; 

•  Intelligence/threat/area  of  operations  expertise;  and 

•  Legal/contracts/administration  expertise. 

As  another  example  of  the  skills  required  for  C2  Assessment  Studies,  the  skills  required  to  provide  OR 
support  to  C2  elements  (such  as  OR/OA  support  to  an  operational  HQ)  is  also  analogous,  as  illustrated  in 
Table  2.1  (RTO,  1999). 

Table  2.1:  Knowledge,  Capabilities  and  Skills  Needed  by  OR/OA  Cell  Team  Members 


Key  Areas 

Knowledge,  Capabilities,  and  Skills  Required 

Analytical 

Probability  theory  Management  tools 

Parametric  and  non-parametric  Organisation  theory 

statistics  Information  management 

Force  and  systems  modelling  Process  engineering 

Traditional  OR  techniques^  Systems  dynamics 

Decision  tools 

Military 

Understand  military  staff  Civil-military  affairs 

Staff  organisation  Situation  appreciation 

Inter-Personal 

Interpersonal  skills  Consultant  skills 

Instructor  skills  Writing  and  presentation  skills 

Software 

Competencies 

Word-processing  Project  management 

Spreadsheets  Mail  systems 

Presentation  Network  systems 

Databases  World  wide  web 

Mapping  Programming  language 

Plus  other  speciality  systems 

Hardware 

Expertise 

Computers  Peripheral  devices 

Printers  Networks 

Scanners 

Linear  programming,  dynamic  programming,  queuing  theory,  inventory  control,  network  analysis  with  PERT,  game  theory 
and  simulation. 
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2.4.2  Background  of  the  Assessment  Team 

Building  a  C2  assessment  team  with  this  full  breadth  of  knowledge,  eapabilities,  and  skill  requires  a  long¬ 
term  commitment  by  the  mother  OR/OA  organisation  to  prepare  a  corpus  of  potential  team  members 
through  recruitment,  education,  training,  and  opportunities  for  appropriate  field  experience. 

Following  the  identification  of  the  skills  required  for  the  team,  those  analysts  made  available  for  the  team 
should  ensure  that  they  leave  a  basic  understanding  of  the  military  fields  under  consideration.  Gaps  in 
experience  should  be  rapidly  filled  through  background  reading,  short  courses,  field  experience, 
or  additional/altemative  analysts  with  the  appropriate  experience  and  skills. 

2.4.3  Forming  the  Assessment  Team 

In  a  study  that  involves  dispersed  and  disparate  organisations  and  teams,  the  effort  to  command  and 
control  the  study  group  must  be  recognised  and  effort  and  time  built  into  the  study  plan.  This  can  be  for 
example,  through  maintenance  of  distributed  working  environments  such  as  web  portals,  information 
campaign  material  and  travel  time  to  meetings.  In  these  cases  the  senior  team  leader  will  revert  to  a  role 
more  akin  to  a  project  manager. 

It  is  one  thing  to  assemble  a  group  of  people,  quite  another  to  forge  them  into  a  coherent  effective 
team.  Sufficient  time  and  a  facilitating  process  should  be  built  in  to  the  project  plan  for  the  group  of 
individuals  to  coalesce  into  a  team. 


2,5  INTERDISCIPLINARY  ASSESSMENT  TEAM  AND  OUTSIDE  RELATIONS 

It  follows  that  C2  analysis,  particularly  for  OOTW  issues,  should  be  done  by  an  interdisciplinary 
assessment  team.  Experienced  analysts  know  that  their  work  owes  success  in  no  small  measure  to  efficient 
working  relationships  within  the  assessment  team  and  with  the  customer  of  analysis.  Building  good 
working  relationships  among  representatives  of  different  scientific  cultures,  such  as  OR/OA  and  IT 
analysts  grounded  in  (hard)  physical  sciences  and  mathematics  on  one  hand  and  (soft)  social  scientists  on 
the  other,  requires  sufficient  mutual  understanding  of  methodologies  and  tools.  In  fact,  differences  in 
scientific  cultures  can  outweigh  differences  in  natural  cultures  provided  that  all  members  of  the 
assessment  team  have  sufficient  command  of  a  common  language.  Therefore,  in  addition  to  leadership  and 
project  management  skills,  the  head  of  the  assessment  team  must  have  a  good  general  idea  of  the  current 
state  of  all  disciplines  involved  in  order  to  compose  an  efficient  team  and  facilitate  interdisciplinary 
co-operation  throughout  the  analysis. 

Good  personal  and  working  relationships  with  the  customer  of  the  analysis  are  essential  for  understanding 
every  aspect  of  the  problem  and  being  able  to  arrive  at  a  problem  structure  and  solution  strategy  that  meets 
the  customer’s  immediate  needs  in  the  light  of  the  strategic  objectives  of  the  respective  OOTW.  Knowing 
the  customer’s  position  in  the  command  hierarchy  and  the  degree  of  influence  he/she  wields  through 
informal  relationships  over  stakeholders  and  actors,  the  co-operation  of  which  might  be  essential  for  an 
implementation  of  analysis  results,  and  understanding  the  respective  consequences  associated  with 
alternative  solutions  is  important  for  assessing  their  acceptability  and  organisational  risk. 

It  is  equally  important  for  the  assessment  team  to  establish  working  relationships  with  the  potential 
subjects  of  study  in  the  early  stages.  This  is  essential  for  capturing  the  nature  and  problem  relevance  of 
formal  and  informal  relations  between  all  organisations,  groups,  and  individuals  that  are  subjects  of  the 
study,  finding  out  about  their  motivations  and  agendas,  and  eliciting  firsthand  information  that  is  critical 
for  solving  the  problem  such  as  their  capabilities  and  the  conditions  attached  for  their  employment. 
However,  the  analyst  should  be  careful  not  to  allow  this  effort  to  gain  greater  understanding  of  the 
problem  to  introduce  bias. 
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2.6  ASSESSMENT  PHASES,  PROCESS,  AND  DYNAMICS 

It  is  important  to  realise  that  all  of  the  elements  of  the  C2  assessment  are  interrelated.  Henee  Problem 
Formulation,  Solution  Strategy,  Measures  of  Merit,  Seenarios,  Human/Organisational  Factors,  Models/ 
Tools/Data,  and  products  are  all  interdependent.  Figure  2.2  illustrates  the  major  phases  and  iterative  nature 
required  for  C2  assessments.  The  Assessment  Process  diagram  was  the  most  difficult  thing  for  the 
SAS026  team  to  agree  upon.  In  essence  this  diagram  is  at  the  heart  of  the  COBP  (Starr,  2001). 
The  remainder  of  this  chapter  discusses  the  key  points  in  this  diagram. 
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Figure  2.2:  C2  Assessment  Process. 


2,6,1  Problem  Formulation 

The  output  of  Problem  Formulation  (Chapter  3)  specifies  the  “what”  of  the  assessment.  C2  studies  tend  to 
be  complex  and  feature  multiple  attributes,  some  of  which  may  not  be  apparent  at  the  start  of  the  study. 
Neither  the  assessment  team  nor  the  sponsor  should  be  surprised  if  the  issues  initially  presented  for  study 
are  replaced  by  other  issues  that  are  closer  to  the  underlying  causes  of  the  initial  problem  or,  in  some 
cases,  symptoms  presented.  A  consequence  of  the  dynamic  nature  of  problem  formulation  is  that  the 
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solution  strategy  and  any  of  the  other  elements  of  the  assessment  may  ehange  as  the  study  progresses. 
Problem  formulation  should  therefore  be  eonsciously  and  routinely  iterated  during  a  study  -  espeeially 
when  new  attributes  start  to  appear.  As  a  minimum  an  iteration  should  oeeur  immediately  following  the 
establishment  of  the  initial  solution  strategy  and  the  assessment  of  study  risk.  Additionally,  the  sponsor 
should  be  aware  that  the  nature  of  the  assessment  team,  sponsor,  or  assumption  provider  teams  might  also 
have  to  expand  or  ehange  with  time.  This  has  implieations  for  planning,  budgeting,  and  tasking. 

In  nearly  all  C2  studies  the  assessment  team  will  study  only  a  subset  of  the  whole  problem  space  due  to 
the  sponsor’s  sphere  of  interest.  This  fact  must  be  recognised  by  the  assessment  team.  An  initial  study  of 
the  complete  problem  space  is  essential  to  establish  this  realisation.  This  will  help  the  assessment  team  to 
understand  the  context  of  the  study  and  provide  advice  to  the  sponsor  on  the  actual  underlying  causes  to 
his  problem  and  consequently  the  requirement  to  involve  other  participants. 

2.6.2  Solution  Strategy 

The  next  step  is  to  develop  a  Solution  Strategy  (Chapter  4)  that  specifies  the  “how”  of  the  assessment. 
Arising  from  the  Solution  Strategy  agreed  upon  and  adopted  by  the  sponsor  are  a  set  of  terms  of  reference 
(e.g.  Statements  of  Work  [SOW]  for  contracts)  that  will  determine  what  work  is  to  be  conducted, 
the  contractual  obligations,  deadlines,  and  resources.  Although  these  must  be  established  as  an 
experimentation  campaign  plan^  and  study  management  plan  (project  plan)  before  work  on  the  project 
begins  in  earnest,  flexibility  must  be  built-in  due  to  the  iterative  nature  of  C2  assessment.  The  assessment 
team  must  be  aware  of  any  preconceived  “solutions”  that  have  been  proposed  by  the  sponsor, 
stakeholders,  and/or  decisionmakers  and  explicitly  deal  with  these  as  appropriate,  avoiding  another 
pressure  to  be  steered  in  a  particular  direction.  The  assessment  team  must  note  if  its  results  are  being 
steered  in  a  particular  direction  and  follow  ethical  behaviour  in  performing  the  study  (see  the  end  of  this 
chapter).  In  many  cases  a  risk-based  approach  to  C2  assessments  can  usefully  complement  the  more 
traditional  cost-effectiveness  approach.  In  particular,  this  helps  decisionmakers  to  deal  with  the  uncertainties  of 
the  real  problem. 

From  a  professional  point  of  view  analysts  should  always  defer  the  selection  of  a  particular  method 
until  the  problem  has  been  formulated  and  a  solution  strategy  has  been  defined.  Recognise  and 
beware  of  “preconceived”  solutions  that  could  influence  the  assessment, 

2.6.3  Review 

Once  there  has  been  a  preliminary  formulation  of  the  problem  and  development  of  a  solution  strategy,  it  is 
imperative  that  an  initial  review  be  conducted.  This  review  should  be  conducted  from  multiple 
perspectives  (e.g.  with  respect  to  the  sponsor’s  initial  problem,  the  feasibility  with  respect  to  resources 
including  team  skills  and  schedule,  soundness  of  the  proposed  analytic  approach).  As  a  result  of  this 
review,  changes  will  usually  be  made  in  both  the  problem  formulation  and  the  solution  strategy. 

2.6.4  Measures  of  Merit,  Scenarios,  and  Human/Organisational  Factors 

At  this  stage  the  assessment  team  must  specify  the  hierarchy  of  Measures  of  Merit  (MoM)  (Chapter  5), 
incorporate  and  identify  relevant  human  and  organisational  factors  (Chapter  6),  and  specify  the 
appropriate  scenarios  (Chapter  7).  As  suggested  by  the  diagram,  there  is  no  unique  sequence  for  doing 
these  tasks.  Iteration  is  required  to  ensure  that  these  tasks  are  done  in  a  coherent,  consistent  fashion. 
When  all  of  these  tasks  have  been  completed,  the  team  has  specified  the  key  variables  to  the  necessary 
level  of  detail  with  adequate  considerations  for  assessment  validity  and  reliability. 


2 

Required  if  the  C2  Assessment  makes  use  of  a  series  of  linked  events  such  as  seminars,  wargames,  command  post  exercises 
(CPX),  field  training  exercises  (FTX),  etc. 
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When  developing  the  MoM  it  is  very  valuable  to  involve  the  sponsor  in  establishing  the  linkages  between 
the  MoMs  and  the  hierarehy  of  MoM.  This  is  beeause  then  the  sponsor  will  then  appreeiate  the  dynamics 
of  the  problem  and  the  requirement  for  breadth  in  the  study.  Although  a  full  set  of  MoM  must  be  derived 
in  accordance  with  the  best  practice  noted  in  Chapter  5,  the  MoM  should  be  prioritised  to  focus  on 
providing  support  to  the  objectives  of  the  study  and  be  practical  and  cost  effective. 

When  selecting  appropriate  scenarios  it  is  good  practise  to  utilise  scenarios  (if  they  exist)  from  a  standard 
set  of  scenarios  approved  for  use  within  the  assessment  and  sponsor  organisation.  The  sponsor  must 
always  be  approached  for  approval  of  the  scenarios.  It  is  bad  practice  to  design  a  scenario  to  prove  a  point. 

2.6.5  Models,  Tools,  and  Data  Requirements 

The  next  step  is  to  iteratively  identify  the  methods  and  tools  (Chapter  8)  and  data  (Chapter  9)  required  to 
perform  the  assessment.  One  of  the  major  challenges  of  the  assessment  is  to  identify  and  gain  access  to 
models,  tools,  and  data  that  are  appropriate  for  exploring  the  issues  of  interest.  The  challenges  come  in 
several  dimensions: 

•  First,  there  is  a  limited  set  of  tools  that  deal  effectively  with  the  C2  dimension  of  the  problem; 

•  Second,  for  even  this  limited  set,  it  is  often  difficult  to  access  and  modify  the  tools  to  reflect  the 
variables  of  interest;  and 

•  Third,  there  is  often  a  paucity  of  useful  data  and  previously  validated  parameters. 

As  a  result  of  the  establishment  of  the  MoM  for  the  study  and  the  data  that  underpins  those  MoM  and 
models,  a  data  collection  and  analysis  plan  should  be  formulated.  The  sponsor  should  also  be  made  aware 
of  the  difficulties  associated  with  getting  appropriate  data,  cost  of  the  data  collection  and  analysis  plan, 
and  the  implications  to  the  study  if  the  required  resources  are  not  set  aside  and  budgeted  to  collect,  collate, 
process,  and  analyse  the  data. 

2.6.6  Assess  Study  Risk 

At  this  point  in  the  process  the  assessment  team  should  take  a  look  at  the  risks  and  uncertainties 
(Chapter  10)  associated  with  the  decisions  made  with  respect  to  all  of  the  tasks  performed  to  date 
(e.g.  consistency  between  the  scenarios  and  the  data,  models  and  availability  of  data,  tools  and  analysis). 
The  sponsor  must  be  made  aware  of  these  risks  and  uncertainties  and  the  strategies  developed  by  the  team 
to  mitigate  them.  If  the  risks  associated  with  the  successful  completion  of  the  study  are  perceived  as  being 
too  high,  the  solution  strategies  should  be  revisited  and  adjusted  accordingly. 

2.6.7  Peer  Review 

When  the  risk  and  uncertainties  are  perceived  as  manageable,  a  peer  review  should  be  conducted. 
Peer  reviews  are  not  used  enough  because  they  tend  to  be  time-consuming,  seen  as  raising  costs, 
or  perceived  as  threatening.  In  addition,  research  teams  often  want  to  perfect  their  results  and  methods 
before  revealing  them.  The  key  is  to  build  a  peer  review  into  the  study  from  the  outset.  The  sponsor  should 
be  informed  as  to  the  importance  of  the  peer  review.  Peer  reviews  should  be  built  into  the  budget  and 
reviewers  invited  to  look  at  the  terms  of  reference,  interim  products,  and  draft  reports  so  that  they  are 
knowledgeable  about  the  effort  and  motivated  to  support  the  project.  In  later  stages  of  the  study,  the  peer 
review  can  improve  presentation  and  also  act  as  a  mechanism  to  make  the  results  known  to  the 
professional  communities.  Over  time  the  assessment  team  should  develop  a  relationship  with  high  quality 
peers  and  use  them  as  a  pool  of  reviewers. 

Peer  reviews  are  not  a  luxury  but  a  necessity. 
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2.6.8  Conduct  of  the  Study 

At  this  point  we  are  in  a  position  to  exeeute  the  assessment.  The  assessment  team  leader  should  keep  a 
study  notebook  or  journal  in  which  all  assumptions  and  decisions  are  documented  so  that  they  are 
available  for  detailed  discussion.  Detailed  administrative  records  need  to  be  kept  regarding  the  data, 
metadata,  models,  and  analytical  and  documentation  tools.  This  will  enable  replication  of  parts  of  the  C2 
analysis  should  the  need  arise.  An  effort  should  be  made  to  create  data  sets  (not  just  the  project  results) 
that  will  be  available  to  other  researchers.  The  resources  required  to  make  such  data  available  to  external 
bodies  needs  to  be  made  clear  to  the  sponsor.  The  conduct  of  the  study  will  not  usually  be  linear.  It  should 
be  anticipated  that  multiple  iterations  will  be  conducted  and  that  lessons  learned  from  initial  data 
collection  and  analysis  efforts  will  inform  subsequent  activities. 

2.6.9  Study  Products 

The  team  must  recognise  the  importance  of  presenting  the  results  of  the  assessment  in  a  clear  and 
comprehensive  manner,  taking  into  consideration  the  style  of  the  decisionmaker  (Chapter  11).  It  is 
particularly  important  that  these  results  illuminate  rather  than  obscure  the  uncertainties  associated  with  the 
assessment. 

2,7  ETHICS 

Professional  operations  research  organisations,  such  as  the  Military  Operations  Research  Society  (MORS), 
have  developed  professional  codes  of  ethics  (Annex  C).  The  assessment  team  should  also  be  guided  by 
a  set  of  professional  ethics  and  standards  of  conduct  to  ensure  the  integrity  and  quality  of  the  analysis. 
This  means  that  the  assessment  team  should,  inter  alia: 

•  Maintain  an  open  and  honest  dialogue  with  the  sponsor  and  other  key  players  within  the  project  in 
order  to  minimise  misunderstandings; 

•  Ensure  that  C2  assessments  are  organised  and  conducted  in  a  balanced  fashion  that  adequately 
identifies  and  represents  all  perspectives,  options,  and  relevant  evidence; 

•  Inform  the  sponsor  and  other  key  players  of: 

•  Any  constraints,  assumptions,  or  circumstances  that  threaten  a  balanced  assessment; 

•  The  risks  and  uncertainties  associated  with  the  methods  and  data  used  in  the  project;  and 

•  Strategies  to  minimise  the  risks. 


2,8  CHAPTER  2  ACRONYMS 


ACE  Resources 
AF(N) 

AMF(E) 

ARRC 

C2 

CDF 

CPX 

FTX 

HB(A) 


Allied  Command  Europe  Resources  -  (Part  of  SHAPE) 
Regional  Command  (North) 

ACE  Mobile  Force  (Eand) 

ACE  Rapid  Reaction  Corps 

Command  and  Control 

Concept  Development  and  Experimentation 

Command  Post  Exercise 

Field  Training  Exercise 

UK  Historical  Branch  (Army) 
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HQ 

Headquarters 

IRTF(L) 

Immediate  Reaction  Task  Force  (Land) 

JCSC 

Joint  Sub-Regional  Command  South  Centre 

JCSE 

Joint  Sub-Regional  Command  South  East 

JFCOM 

US  Joint  Forces  Command 

KIBOWI 

NL  Army  Exercise  Driver 

MND(C) 

Multinational  Division  (Central) 

MoM 

Measures  of  Merit 

MORS 

Military  Operations  Research  Society 

NC3A 

NATO  C3  (Consultation,  Command  &  Control)  Agency 

NL  MOD 

Netherlands  Ministry  of  Defence 

OOTW 

Operations  Other  Than  War 

OA 

Operational  Analysis 

OR 

Operations  Research 

PRL 

SHAPE  Policy  Requirements  Land 

SACLANT  OA 

Supreme  Allied  Command  Atlantic  Operational  Analysis  Cell 

SFS 

Strike  Force  South 

SHAPE 

Supreme  HQ  Allied  Powers  Europe 

SOW 

Statements  of  Work 

WPC 

Warrior  Preparation  Center  (Ramstein  Germany) 
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Appendix  1  to  Chapter  2  - 

PARTICIPANT  MAPPING  OF  THE  EVALUATION  OF 
THE  IMMEDIATE  REACTION  TASK  FORCE  (LAND) 
C2  CONCEPT  -  AN  EXPLANATION  OF  FIGURE  2.1 


2A.1  BACKGROUND 

The  Immediate  Reaction  Task  Force  (Land)  (IRTF(L))  command  and  control  concept  was  proposed  in 
1998  as  a  mechanism  to  modernise  the  ACE  Mobile  Force  (Land)  (AMF(L)).  The  IRTF(L)  concept  is 
predicated  on  the  enlargement  of  AMF(L)  from  brigade  size  up  to  division  size  with  a  single  streamlined 
headquarters  and  a  chain  of  command  using  embedded  mini-Task  Group  HQ  cells.  This  was  evaluated 
between  1999  and  2001  as  a  test  case  for  the  NATO  Concept  Evaluation  and  Experimentation  (CDE) 
process  using  a  series  of  FTX,  CPX,  wargames,  simulations  and  historical  analyses. 


2A.2  ASSESSMENT  TEAM 

In  the  case  of  the  IRTF(L)  study  the  assessment  team  was  led  by  NCSA  OR  Division,  with  contracted 
experts,  analytical  and  military  support  from  KS  Consultants  and  UK  DERA.  Free  analytical  support  was 
also  made  available  at  peak  periods  from  US  JFCOM  and  SACLANT  Operational  Analysis  (such  as  the 
exercises  of  the  experimentation  campaign).  The  sponsor  was  SHAPE  Policy  and  Requirements  Land  - 
who  were  tasked  with  the  evaluation  of  the  C2  Concept. 

It  was  clear  to  the  assessment  team  as  to  why  the  sponsor  wanted  the  study  -  a  straightforward  evaluation 
of  the  military  utility  (to  NATO)  of  the  C2  Concept.  However  at  the  end  of  the  study  the  results  were 
combined  with  other  issues,  and  decisions  were  made  on  the  future  of  the  unit  under  study.  This  was 
something  that  was  not  foreseen  by  any  of  the  participants  at  the  start  of  the  project. 


2A.3  DECISIONMAKERS  OR  PROBLEM  OWNERS 

The  sponsor’s  task  was  to  provide  advice  up  the  chain  of  command  to  SACEUR  and  ultimately  the 
Military  Committee  on  the  efficacy  of  the  C2  Concept.  Although  the  HQ  ACE  Mobile  Force  (Land) 
was  the  subject  of  the  study  it  was  also  party  to  any  decision  regarding  its  own  future  modernisation.  It  is 
commanded  directly  by  SACEUR  via  SHAPE. 

SACLANT  CDE,  however,  was  not  in  the  command  chain,  but  was  seen  as  a  decision  maker  within  the 
context  of  the  study  as  it  was  interested  in  the  experience  of  the  team  in  conducting  the  study  as  a  test  case 
to  illustrate  the  value  of  NATO  Centred  CDE  to  the  Alliance. 


2A.4  STAKEHOLDERS 

The  ACE  Mobile  Force  (Land)  was  the  main  stakeholder  as  it  was  the  subject  of  the  study.  As  a  decision 
maker,  data  and  assumption  provider  and  also  possible  member  of  a  future  study  team  it  was  in  a  very 
powerful  position,  and  was  approached  and  treated  with  much  respect  by  the  assessment  team.  After  a 
shaky  start  (where  neither  side  was  sure  of  the  other’s  intentions)  a  good  working  relationship  was 
established  over  the  period  of  the  project. 

The  Netherlands  MOD  -  in  the  form  of  the  Royal  Netherlands  Army  -  was  the  provider  of  the  Command 
Information  System  (ISIS)  used  as  the  digitisation  vehicle  for  the  evaluation  of  the  concept.  As  such  it  was 
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directly  affected  by  the  exercise  program  used  for  the  experimentation  and  also  the  future  of  the  concept 
and  AMF(L). 

The  Military  Committee  was  also  a  stakeholder,  representing  the  Nations  of  NATO  that  contribute  troops 
and  staff  to  the  AMF(L),  and  these  nations  would  be  directly  affected  by  any  decision  on  the  concept. 

ACE  Resources  at  SFIAPE  were  also  a  stakeholder  as  they  were  required  to  sanction  and  organise  any 
manning  changes  proposed  for  the  FIQ  -  including  the  temporary  additional  manning  required  for  the 
evaluation. 


2A.5  BILL  PAYER 

Monies  were  mostly  provided  from  the  slice  of  the  NCSA  Scientific  Program  Of  Work  (SPOW)  controlled 
by  SFIAPE  PRE.  In  the  initial  stages  of  the  project  additional  monies  were  also  provided  by  SACEANT. 
Monies  also  had  to  be  sought  from  the  SHAPE  Exercise  budget  to  pay  for  movement  of  the  exercise 
observers  in  order  to  attend  the  exercises. 

Throughout  the  project  the  NCSA  was  the  contractual  authority  to  let  contracts  on  behalf  of  the  bill  payer. 


2A.6  EXISTING  STUDY  TEAMS 

An  extensive  literature  search  was  conducted  for  the  study  -  with  the  majority  of  recent  references 
occurring  within  the  UK  and  US.  Exploratory  trips  (organised  through  US  JFCOM)  to  US  Battle  labs  and 
UK  facilities  (through  UK  DERA)  revealed  the  current  state  of  knowledge  with  respect  to  measuring  C2 
performance  in  exercises  and  evaluating  new  C2  concepts.  In  response  to  this  the  data  collection 
methodology  was  based  initially  on  the  Fort  Eeavenworth,  US  Army  Research  Institute  ACCES  method. 


2A.7  FUTURE  STUDY  TEAMS 

It  was  identified  at  an  early  stage  that  there  could  be  future  related  projects.  In  particular  those  relating  to 
expeditionary  and  initial  entry  forces.  The  probable  NATO  organisations  that  could  be  involved  in  such 
studies  were  NCSA,  SHAPE  PRE,  AMF(E),  Multinational  Division  (Central),  Strike  Force  South  and  the 
ACE  Rapid  Reaction  Corps.  Of  course  there  would  probably  be  future  study  teams  within  the  nations  - 
but  these  plans  are  not  visible  to  NATO.  Consequently,  as  the  assessment  team  was  very  likely  to  be 
involved  in  such  future  work;  all  data  was  archived  and  routinely  written  up  and  published. 


2A.8  PEER  REVIEWERS 

The  assessment  team  were  not  able  to  arrange  a  formal  Peer  Review  of  the  solution  strategy  adopted. 
This  mechanism  does  not  yet  exist  for  NATO  centred  studies.  What  was  achieved  the  submission  of  the 
problem  to  the  SAS026  panel  as  an  example  for  testing  the  coverage  of  the  revised  COBP.  This  yielded 
some  practical  advice  and  helped  the  assessment  team  better  understand  the  dynamics  of  the  project. 


2A.9  DATA  PROVIDERS 

Most  of  the  data  for  the  evaluation  were  derived  from  NATO  training  exercises  run  by  or  for  AMF(E), 
MND(C),  SFS  and  Joint  Sub  Regional  Commands  South  East  and  South  Centre.  In  all  cases  relationships 
had  to  be  curried  by  the  assessment  team  and  sponsor  to  allow  access  to  the  HQ  and  Exercise  Control  for 
the  exercise  observers,  and  for  background  materiel.  In  two  cases  national  exercise  training  centres 
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and  exercise  drivers  were  hired  by  the  assessment  team  to  support  command  post  exercises 
(Warrior  Preparation  Center,  and  the  KIBOWI  exercise  driver).  Historical  data  for  the  study  was  also 
provided  from  the  UK  Historical  Branch  (Army)  -  which  was  approached  via  the  contracted  UK  members 
of  the  assessment  team. 


2A.10  ASSUMPTION  PROVIDERS 

The  assessment  team  was  in  the  fortunate  position  to  actually  be  one  of  the  assumption  providers  - 
through  NCSA’s  and  the  sponsor’s  involvement  in  the  NATO  Defence  Requirements  Review.  The  owner 
of  the  C2  Concept  however  remained  HQ  AMF(L)  itself,  and  therefore  remained  the  authority  as  to  its 
conceptual  and  physical  implementation. 


2A.II  DATA  COLLECTORS 

In  the  case  of  the  IRTF(L)  study,  data  was  largely  extracted  through  observation  of  HQ  activities  during 
exercises  and  team-in-the-loop  wargames.  In  all  of  these  exercises  Subject  Matter  Experts  (SME)  were 
used  to  observe  functional  and  cross  functional  activities  in  the  HQ.  Most  of  the  military  SMEs  were 
provided  by  Regional  Command  AF  NORTH  and  its  subordinate  commands  across  Allied  Command 
Europe  (ACE).  Additional  data  collectors  were  also  provided  by  the  German  University  of  the  Federal 
Armed  Forces  and  US  JFCOM.  UK  DERA  provided  military  analysts  to  lead  some  of  the  activities 
involved  in  capturing  the  HQ  processes. 
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“First  find  out  what  the  question  is  -  then  find  out  what  the  real  question  is.  ”  —Vince  Roske 


3.1  DEFINITION  OF  PROBLEM  FORMULATION 

Effective  problem  formulation  is  fundamental  to  the  success  of  all  analysis,  but  particularly  in  Command 
and  Control  (C2)  assessment  because  the  problems  are  often  ill-defined  and  complex,  involving  many 
dimensions  and  a  rich  context.  Problem  formulation  involves  decomposition  of  the  analytic  problem  into 
appropriate  dimensions  such  as  structures,  functions,  mission  areas,  command  echelons,  and  C2  systems. 
Problem  formulation  is  an  iterative  process  that  evolves  over  the  course  of  the  study.  It  is  essential  even 
for  small  studies  or  where  time  is  short  -  it  will  save  time  later  and  help  ensure  quality. 

The  problem  formulation  phase  should  identify  the  context  of  the  study  and  aspects  of  the  problem- 
related  issues. 

The  context  of  the  study  includes: 

•  Geopolitical  context  that  bounds  the  problem  space; 

•  Political,  social,  historical,  economic,  geographic,  technological  environments; 
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•  Actors; 

•  Threats; 

•  Aim  and  objectives  of  the  analysis,  including  the  decisions  to  be  supported; 

•  Generic  C2  issues'; 

•  Relevant  previous  studies;  and 

•  Stakeholders  and  their  organisational  affiliation  (including  both  stakeholders  of  the  problem  and 
stakeholders  of  the  study). 

The  aspects  of  the  problem  include: 

•  Issues  to  be  addressed; 

•  Assumptions; 

•  High-level  Measures  of  Merit  (MoM); 

•  Independent  variables  (controllable  and  uncontrollable); 

•  Constraints  on  the  values  of  the  variables  (domain  and  range); 

•  Time  constraints  on  delivery  of  advice  to  the  decisionmaker;  and 

•  Whether  this  is  a  single  decision  or  (possibly  one  of)  a  chain  of  decisions  to  be  made  over  time. 

The  problem  is  not  formulated  until  the  assessment  team  has  addressed  each  aspect  of  the  problem. 

In  simple  terms,  problem  formulation  can  be  seen  as  an  iterative  process.  First,  the  team  must  identify  the 
variables  that  bound  the  problem  space.  Then  they  must  determine  which  of  these  are  outputs  (dependent 
variables)  and  which  of  these  are  inputs  (independent  variables).  The  team  proceeds  by  iterating  to  build 
an  understanding  of  how  these  relate  to  each  other.  It  should  be  viewed  as  a  voyage  of  discovery.  In  most, 
if  not  all,  cases  of  C2  assessment,  the  knowledge  domain  under  study  is  in  fact  a  system  characterised  by 
rich  interaction  and  feedback  among  all  the  factors  or  variables  of  interest.  The  choice  of  dependent 
variables  results  from  a  clear  specification  of  the  issues  and  products  needed  to  satisfy  the  terms  of 
reference.  Independent  and  intervening  variables  are  also  chosen  based  on  the  purpose  of  the  analysis. 

In  the  initial  problem  formulation  iteration,  it  is  critical  to  begin  with  an  understanding  of  the 
REAL  problem  rather  than  a  determination  to  apply  readily  available  tools,  scenarios,  and  data. 

Within  the  NATO  context,  a  number  of  documents  are  available  or  under  development  that  may  assist  in 
understanding  the  study  context.  They  are  listed  at  the  end  of  this  chapter. 


3.2  PRINCIPLES  OF  PROBLEM  FORMULATION 

There  is  no  universally  acceptable  approach  to  problem  formulation.  However,  best  practices  exist  that  can 
be  applied.  The  principles  associated  with  problem  formulation  are  addressed  in  two  categories:  those  that 
are  appropriate  for  all  C2  assessments  and  those  that  are  appropriate  for  assessments  of  C2  for  Operations 
Other  Than  War  (OOTW). 


'  Broad  C2  issues  include  key  systems,  doctrine,  Tactics,  Techniques,  and  Procedures  (TTP),  organisational  structures,  and  key 
assumptions  (e.g.  system  performance  parameters). 
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3.2.1  Principles  Appropriate  for  C2  Assessments 

Explicit  problem  formulation  must  precede  construction  of  concepts  for  analysis  or  method  selection. 
This  is  not  a  trivial  exercise,  especially  in  C2  assessments.  Proper  resourcing  of  problem  formulation 
activities  will  improve  the  overall  efficiency  and  quality  of  the  study. 

An  understanding  of  the  decisions  to  be  supported  by  the  analysis  and  the  viewpoints  of  the  various 
stakeholders  (e.g.  customers,  users,  and  suppliers)  is  essential  to  clarifying  the  study  issues. 
This  understanding  should  be  fed  back  to  the  stakeholders.  A  careful  review  of  previous  and  current  work 
must  be  carried  out  as  a  valuable  source  of  ideas,  information,  and  insight.  This  review  should  also  serve 
to  identify  pitfalls  and  analytic  challenges. 

Problem  formulation  must  not  only  provide  problem  segments  amenable  to  analysis,  but  also  a  clear 
and  valid  mechanism  for  meaningful  synthesis  to  provide  coherent  knowledge  about  the  original, 
larger  problem.  The  formulated  problem  is  thus  an  abstraction  of  the  real  problem  that  can  be  defined  in 
terms  of  dependent  variables  that  relate  to  this  real  problem  and  coherent  settings  for  the  independent 
variables  that  can  be  interpreted  in  terms  of  decisions  and  actions  by  the  customer. 

Problem  formulation  must  be  broad  and  iterative  in  nature,  accepting  the  minimum  of  a  priori  constraints 
and  using  methods  to  encourage  creative  and  multi-disciplinary  thinking,  such  as  proposing  a  number  of 
hypotheses  for  the  expression  of  the  problem.  It  must  be  recognised  that  change  is  inevitable  in  many 
dimensions  (e.g.  understanding  of  the  problem,  requirements,  technologies,  co-evolution  of  concepts  of 
operation,  command  concepts,  organisation,  doctrine,  systems).  Thus  the  assessment  process  must 
anticipate  and  accommodate  this  change. 

Practical  constraints  such  as  data  availability,  study  resources  (including  time),  and  limitations  of  tools 
should  be  treated  as  modifiers  of  the  problem  formulation  rather  than  initial  drivers.  Such  constraint  may, 
in  the  end,  drive  the  feasible  solutions,  but  it  is  important  to  recognise  this  as  a  compromise  rather  than  an 
ideal.  Proper  problem  formulation  takes  substantial  time  and  effort! 

It  is  important  that  problem  formulation  address  risk  from  multiple  perspectives.  In  addition  to  sensitivity 
analysis  of  the  dependent  variables,  risk  analysis  techniques  should  be  used  to  directly  explore  options  to 
mitigate  risk  (Chapter  10). 

C2  assessment  often  involves  impacts  on  defence  business  outside  the  context  of  a  particular  campaign  or 
operation.  The  study  must  address  these  impacts. 

3.2.2  Principles  Appropriate  for  OOTW  C2  Assessments 

Problem  formulation  must  address  the  geopolitical  context  of  the  problem  and  seek  to  identify  the  “broad” 
C2  issues  contained  within  the  terms  of  reference  for  the  study.  There  are  no  universal  societal  “norms”. 
Therefore,  care  must  be  taken  in  attempting  to  transfer  the  experience  in  one  OOTW  to  another. 

OOTW  C2  assessments  often  involve  policy-related  impacts  outside  the  context  of  a  particular  military 
operation.  Therefore,  MoM  hierarchies  must  contain  measures  of  policy  effectiveness. 

An  historical  perspective  is  critical  to  understanding  OOTW  because  social  conflict  and  structures  often 
have  roots  far  back  in  history.  However,  it  must  be  remembered  that  present-day  social  behaviour  is  not 
driven  by  historical  events  themselves,  but  by  present-day  perceptions,  processes,  and  prejudices  which 
have  evolved  from  the  past. 

A  key  risk  in  complex  OOTW  studies  is  allowing  the  problem  formulation  process  to  focus  prematurely 
on  subsets  of  the  problem  because  they  are:  a)  interesting;  b)  familiar;  c)  pre-judged  to  be  critical; 
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or  d)  explicitly  called  out  by  the  customer.  This  requires  great  discipline  by  the  study  team,  especially 
where  the  team’s  previous  experience  is  biased  in  favour  of  particular  parts  of  the  problem  space. 

The  assessment  team  needs  access  to  subject  matter  experts  from  a  broad  range  of  disciplines 
(e.g.  social  scientists,  historians,  and  regional  experts  in  OOTW  assessment). 


3.3  PROBLEM  FORMULATION  PROCESS 

During  the  early  stages  of  problem  formulation  it  is  important  to  quickly  cover  the  whole  problem  and 
produce  an  initial  formulation  (i.e.  an  explicit  expression  of  the  problem).  See  Figure  3.1.  This  prevents 
premature  narrowing  of  the  assessment  and  serves  as  an  aid  to  shared  situation  awareness  within  the  study 
team. 


Formulated  Problem 


Contexr 

of  the  Study 


Problem 
Formulation 
Tools 


Issues  to  be  addressed 
Assumptions 
High  Level  MoM 
Independent  Variables 

-  Controllable 

-  Uncontrollable 
Constraints  on  Variables 


Figure  3.1:  The  Formulated  Problem. 


The  process  begins  with  the  sponsor  presenting  the  assessment  team  with  a  problem  to  assess  and  an 
articulation  of  broad  constraints  (e.g.  schedule,  resources).  Based  on  a  preliminary  assessment  of  the 
problem,  the  team  identifies  the  key  issues  to  address.  This  identification  of  key  issues  leads  to  a 
characterisation  of  the  context  for  the  study  (e.g.  relevant  geopolitical  factors,  identification  of  the  key 
actors  and  threats,  identification  of  generic  C2  issues,  review  of  prior  studies).  Based  on  the  results  of  this 
characterisation,  the  analysis  team  identifies  what  it  perceives  as  the  real  issues  to  address.  It  is  vital  for 
the  team  to  engage  in  a  dialogue  with  the  key  sponsor  and  stakeholders  to  get  “buy  in”  for  these  issues. 
Once  that  is  achieved,  the  team  must  identify  and  characterise  the  remaining  elements  of  the  problem 
formulation  phase.  To  facilitate  that  activity,  the  analysis  team  should  identify/create  and  apply  selected 
problem  formulation  tools  and  techniques  (e.g.  brainstorming,  Delphi  analyses,  directed  graphics, 
influence  diagrams).  The  results  of  that  activity  will  include  a  summary  of  the  assumptions,  high-level 
MoM,  independent  variables  (both  controllable  and  uncontrollable),  and  constraints  on  the  variables. 
Once  it  is  co-ordinated  with  the  sponsor  and  stakeholders,  the  end  product  documents  what  is  to  be  done 
in  the  analysis.  The  next  key  activity  will  be  to  develop  a  solution  strategy  that  describes  how  the  study  is 
to  be  done. 
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3.3.1  Bounding  the  Problem/Issues  and  Assumptions 

In  dealing  with  fuzzy  or  uncertain  boundaries,  the  problem  formulation  process  needs  to  explore  and 
understand  the  significance  of  each  boundary  before  making  (or  seeking  from  customers)  assumptions 
about  it.  This  involves  keeping  an  open  mind,  during  the  early  stages  of  problem  formulation,  about  where 
the  boundaries  lie  and  their  dimensional  nature.  This  is  difficult  because  it  makes  the  problem  modelling 
process  more  complicated.  A  call  for  hard  specification  too  early  in  the  problem  formulation  process  must 
be  avoided.  In  the  end,  of  course,  the  problem  must  be  formulated  in  order  to  solve  it,  but  formulation 
should  be  an  output  from  the  first  full  iteration,  not  an  early  input  to  it. 

In  formulating  an  OOTW  problem,  we  are  trying  to  bound  a  complex  system.  This  is  partly  a  process  of 
understanding  boundaries  which  exist  in  reality  (e.g.  mission  statements,  geographical  areas  and  the 
timing  of  a  procurement  process)  and  partly  imposing  artificial  boundaries  in  order  to  illuminate  the 
structure  of  the  problem  and  constrain  the  scope  of  the  analysis.  To  avoid  the  trap  of  over-specification, 
boundaries  (especially  self-imposed  ones)  should  be  kept  porous,  allowing  for  cause  and  effect  chains  to 
flow  through  the  external  environment  of  the  portion  of  the  complex  system  that  the  boundaries  define. 

While  clear  definitions  and  hard  conceptual  boundaries  are  ultimately  necessary  in  order  to  create  a 
manageable  problem  space,  care  must  be  taken  to  avoid  coming  to  closure  prematurely. 

3.3.2  High-Level  MoM 

Identification  of  high-level  MoM  should  start  with  ideal  measures  of  the  desired  benefits  before 
considering  what  can  be  practically  generated  by  analysis  (the  latter  may  force  the  use  of  surrogate  MoM, 
but  these  must  be  clearly  related  to  the  desired  measures). 

A  structured  analysis  of  potential  benefits^  should  be  carried  out  as  a  basis  for  constructing  appropriate 
MoM.  Mapping  techniques,  such  as  cognitive  and  causal  mapping,  are  a  good  way  to  express  the  various 
relationships  within  the  problem  space  and  to  identify  ‘chains’  of  analysis  (i.e.  links  among  the 
independent  variables  and  between  the  independent  and  dependent  variables).  These  lead  to  resultant 
structure  in  terms  of  independent  and  dependent  variables,  and  hence  to  high-level  MoM. 

3.3.3  Prohlem  Formulation  Tools 

It  is  useful  to  identify,  develop  (if  necessary),  and  apply  appropriate  tools  to  support  problem  formulation. 
Representative  tools  and  techniques  include:  techniques  for  supporting  expert  elicitation,  influence 
diagrams,  causal  maps,  system  dynamic  models,  and  agent-based  models. 

Problem  Formulation  is  fundamentally  a  social  process  of  developing  a  shared  understanding.  People 
skills  such  as  the  ability  to  facilitate  a  ‘brainstorming  session’  or  to  elicit  information  and  context,  are  thus 
important.  ‘Throwaway  models’  (which  may  be  simple  simulation  models,  causal  maps,  system  dynamic 
models,  etc.)  may  be  developed  as  part  of  the  process,  and  then  discarded  as  insight  is  gained. 

Tools  and  approaches  used  for  problem  formulation  must  be  consistent  with  other  tools  and  techniques 
likely  to  be  considered  for  the  subsequent  analysis  in  order  to  produce  a  sensible  ‘multi-methodology’ 
approach  to  the  entire  problem  and  its  solution. 


3,4  CONSTRAINTS  ON  THE  VARIABLES 

The  formulation  of  the  problem  is  completed  when  the  constraints  on  either  the  independent  or  dependent 
variables  have  been  identified.  Constraints  on  the  dependent  variables  represent  “acceptable”  thresholds  or 

2 

The  structured  analysis  of  benefits  is  a  logical  process  that  seeks  causally  to  map  lower  level  MoM  that  can  be  related  to 
investments  or  other  actions  to  higher  level  MoM  that  can  be  valued  directly  by  decisionmakers. 
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limits.  For  example,  one  could  place  a  constraint  on  blue  loss,  time  to  accomplish  a  mission,  collateral 
damage,  or  some  combination  of  factors.  Constraints  on  the  independent  variables  represent  either  feasible 
or  acceptable  limits  on  such  factors  as  human  performance,  C2  system  performance,  or  even  supplies. 
They  also  could  represent  doctrinal  or  legal  processes  that  act  as  constraints. 

3,4,1  The  Next  Step 

The  next  step  in  the  C2  assessment  process  is  the  development  of  a  solution  strategy.  It  should  be  noted 
that  the  team  is  not  finished  with  problem  formulation  at  this  point  but  is  now  ready  to  proceed  to  build  a 
solution  strategy.  As  work  progress  on  the  development  of  a  solution  strategy,  it  will  also  certainly  be 
necessary  to  revisit  the  specification  of  high-level  MoM  and  the  constraints.  This  chapter  concludes  with  a 
discussion  of  the  products  of  problem  formulation. 


3,5  PRODUCTS  OF  PROBLEM  FORMULATION 

Figure  3.2  depicts  the  essential  elements  of  the  formulated  problem. 


A  checklist  can  be  used  to  ensure  that  all  the  aspects  described  in  the  definition  have  been  covered. 
These  include: 

•  Precise  statements  of  the  question  being  researched; 

•  A  list  of  independent  variables; 

•  A  list  of  high-level  MoM;  and 

•  A  list  of  assumptions  and  constraints. 
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3.5.1  Diagrams 

Typically,  the  problem  formulation  phase  should  also  produce  a  number  of  diagrams  such  as  influence 
maps  which  summarise  the  key  issues  and  interactions. 

3.5.2  Data  Glossary 

The  problem  formulation  phase  must  begin  to  create  a  glossary  of  key  data  elements,  metadata, 
information,  and  terms. 


3,6  CHAPTER  3  ACRONYMS 


C2  Command  and  Control 

MoM  Measures  of  Merit 

OOTW  Operations  Other  Than  War 

TTP  Tactics,  Techniques,  and  Procedures 
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‘‘Operations  research  is  a  scientific  method.  Executives  have  often  in  the  past  used  some  of  the  techniques 
...to  help  themselves  arrive  at  decisions  ...  But  the  term  “scientific  methods’’  implies  more  than  sporadic 
application  and  occasional  use  of  a  certain  methodology;  it  implies  recognised  and  organised  activity 
amenable  to  application  to  a  variety  of  problems  and  capable  of  being  taught.  ’’  -Philip  M.  Morse  and 
George  E  Kimball,  Methods  of  Operations  Research 


4.1  THE  STUDY  PLAN 

A  conscientious  effort  is  required  to  create  and  follow  a  study  plan  that  guides  data  collection  and  analyses 
and  prepares  for  the  use  of  the  insights  and  data  to  be  collected  to  contribute  to  a  solution  to  the  problem  at 
hand.  The  study  plan  consists  of  two  inter-related  parts  -  the  formulated  problem  {the  What)  and  the 
solution  strategy  {the  How).  The  output  of  the  initial  problem  formulation  provides  the  assessment  team 
with  an  operating  definition  of  what  needs  to  be  done.  The  output  of  the  solution  strategy  phase  provides 
the  team  with  an  operating  definition  of  how  this  will  be  accomplished.  As  the  project  unfolds,  there  will 
usually  be  a  significant  amount  of  iteration  that  both  modifies  the  problem  formulation  and  the  solution 
strategy.  Without  a  study  plan,  it  is  unlikely  that  needed  efforts  will  be  properly  scoped,  prioritised, 
scheduled,  and  resourced.  Even  if  the  way  ahead  seems  clear,  the  articulation  of  a  formal  Solution  Strategy 
is  necessary. 
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The  objective  of  this  phase  of  the  study  is  to  develop  a  feasible  approach  to  go  from  the  specification  of 
what  is  to  be  done  to  how  it  is  to  be  done.  This  involves  developing  an  approach  that  will  result  in  the 
team’s  ability  to  collect  the  data  necessary  to  determine  the  values  of  the  Measures  of  Merit  (MoM) 
for  specified  values  of  independent  variables.  The  characteristics  of  data  collection  instruments  and 
analysis  tools  and  techniques  will  determine  the  resources  required,  the  time  needed,  and  the  risks  inherent 
in  the  solution  approach.  When  compared  to  the  study  constraints  and  the  problem  formulation,  it  will  be 
determined  whether  the  solution  approach  is  both  feasible  and  satisfies  the  requirements  of  the  problem 
formulation  (e.g.  measures  the  right  things). 

Figure  4. 1  depicts  what  is  involved  in  moving  from  a  problem  formulation  to  a  solution  strategy. 


Figure  4.1:  From  Problem  Formulation  to  Solution  Strategy. 


4.2  KEY  DEFINITIONS 
4,2,1  Solution  Strategy 

A  solution  strategy  consists  of  the  specification  of  a  set  of  sequential  and  parallel  analytical  steps, 
often  involving  several  methodologies  and  tools.  The  solution  strategy  is  designed  to  begin  with  what  is 
known,  and  by  execution  of  the  specified  steps,  leads  to  what  one  desires  to  know  -  an  illumination  of  the 
issues.  The  strategy  can  be: 

•  Simple  -  calculate  mortgage  payments  by  finding  the  input  values  for  the  payment  equation  and 
then  evaluating  the  result; 

•  Moderately  complicated  -  define  input  variables,  output  variables  and  precision  requirements, 
create  a  designed  experiment  and  run  the  experiment  with  appropriate  measurements  including  the 
regression  analysis;  or 

•  Extremely  complex  -  identify  the  relevant  variables  and  systems  of  variables,  specify  how  they 
might  be  measured,  hypothesize  how  they  are  related,  and  design  research  strategies  that  allow  for 
complex  adaptive  systems  or  other  “messy”  structures  or  processes. 
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The  solution  strategy  must  take  the  outputs  from  problem  formulation,  refine  and  operationalise  them, 
and  develop  a  plan  to  collect  and  analyse  appropriate  data  (including  the  development  and/or  selection  of 
models,  the  design  of  collection  instruments,  and  the  selection  of  analysis  tools)  to  understand  the 
relationships  among  the  relevant  variables  associated  with  MoM,  the  scenarios,  and  human  and 
organisational  factors  (Figure  4.2). 


Figure  4.2:  Solution  Strategy. 


4.2.2  Measures  of  Merit  (MoM) 

MoM  are  a  set  of  variables  that  focus  the  assessment  on  the  issues  of  interest.  In  most  analyses,  these  are 
the  dependent  variables.  In  many  cases  there  are  significant  inter-relationships  among  the  MoM. 

4.2.3  Human  Factors 

Human  factors  consists  of  a  set  of  variables  that  characterise  concepts  including  beliefs,  cultural  norms, 
stress,  fatigue,  fear,  arousal,  morale,  intelligence,  and  level  of  experience.  In  Command  and  Control  (C2) 
assessments  these  are  typically  independent  or  intervening  variables. 

4.2.4  Organisational  Factors 

Organisational  factors  consist  of  a  set  of  variables  that  characterise  organisations,  such  as  cohesion, 
command  structure,  explicit  and  tacit  relationships,  information  flows,  and  organisational  cultures. 
These  are  also  typically  independent  of  intervening  variables  in  C2  analyses. 

4.2.5  Scenarios 

Scenarios  consist  of  the  evolution  in  time  of  several  elements:  a  context  (e.g.  a  characterisation  of  a 
geopolitical  situation),  the  participants  (e.g.  intentions,  capabilities  of  blue,  red,  others),  and  the 
environment  (e.g.  natural,  weather  and  manmade,  mines).  In  C2  assessments,  the  purpose  of  scenarios  is  to 
ensure  that  the  analysis  is  performed  within  the  appropriate  range  of  opportunities  to  observe  the  relevant 
variables  and  their  interrelationships. 
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4.2.6  Model 

A  model  is  a  physical,  analogue,  or  symbolic  representation  of  relevant  aspects  of  reality  for  a  purpose. 
It  is  an  abstraction  of  reality.  A  model  emphasises  particular  aspects  (a  subset)  of  reality. 

The  assessment  model  of  primary  interest  is  the  assessment  team’s  model  (conceptual  or  in  some  analytic 
manifestation)  of  the  C2  problem  including  the  variables  of  interest,  their  hypothesised  relationships, 
and  any  prior  assumption  about  their  values  and  linkages.  The  assessment  team  may  also  employ  or 
develop  other  models  or  simulations  in  order  to  perform  analysis  or  explore  risks  and  uncertainties. 
Some  teams  will  employ  more  than  one  of  these  analytic  tools. 

4.2.7  Tool 

A  tool  facilitates  the  exploration  of  relationships  among  model  variables  and/or  develops  “solutions” 
(e.g.  maximise  value  subject  to  constraints).  A  tool  may  be  as  simple  as  a  checklist  or  an  algorithm,  or  it 
may  be  an  extremely  large  simulation.  A  simulation  is  the  instantiation  of  a  model  that  serves  to  facilitate 
the  exploration  of  the  relationships  among  the  variables  -  it  generates  data  for  analysis  and  generally 
emphasises  the  passage  of  time.  Models  and  simulations  are  frequently  subdivided  into  categories  of 
constructive,  virtual,  and  live. 

Occasionally,  the  distinctions  among  a  tool,  model,  and  data  are  subtle.  For  example,  in  a  linear  program 
the  model  is  the  set  of  formulas  that  specify  the  objective  function  and  the  constraints.  The  tool  is  the 
simplex  method  (or  similar  algorithmic  solution  method).  The  data  is  an  instantiation  of  the  formulas 
(provides  values  for  the  coefficients  and  constants).  In  the  case  of  a  simulation,  the  simulation 
environment  and  simulation  engine  are  the  tools,  the  coded  simulation  embodies  the  model,  and  the  input 
values  to  the  simulation  comprise  the  data.  Often,  the  simulation  code  and  the  data,  together,  are  required 
for  a  complete  definition  of  the  model. 

4.2.8  Data 

Data  are  the  values  associated  with  the  variables.  Data  may  be  ratio,  interval,  ordinal,  or  nominal  in  scale. 
Data  may  originate  from  empirical  observation;  be  derived  from  models,  simulations,  or  analyses; 
be  established  from  subject  matter  experts;  or  be  established  by  assumption. 


4.3  DEVELOPING  A  SOLUTION  STRATEGY 

The  development  of  a  solution  strategy  is  an  iterative  process  that  strikes  an  artful  balance  between 
what  the  team  would  like  to  do  and  what,  given  the  state  of  the  art,  the  available  data,  tools,  schedule, 
and  resources,  is  possible  to  do. 

4.3.1  Prerequisites 

The  solution  strategy  should  not  be  designed  before  the  problem  formulation  process  is  substantially 
complete  (see  Fig.  4.1)  and  the  problem  formulation  products  specified  in  Chapter  3  are  available  to  the 
team.  This  means  that: 

•  The  “real”  question  to  be  answered  is  known; 

•  The  assumptions  have  been  articulated; 

•  The  high  level  MoM  have  been  identified; 

•  The  independent  variables  have  been  identified;  and 

•  The  constraints  associated  with  the  variables  have  been  identified. 
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However,  the  assessment  team  should  always  remember  the  inherently  iterative  nature  of  the  process. 
Adjustments  may  prove  necessary  in  the  problem  formulation  as  the  solution  strategy  matures. 

4,3,2  Steps  in  Developing  a  Solution  Strategy 

As  an  initial  step,  the  team  should  elaborate  on  the  MoM  to  specify  the  detailed  MoM  that  are  to  be 
evaluated.  This  is  sometimes  referred  to  as  developing  operational  definitions  for  the  MoM  -  definitions 
that  specify  the  metric  to  be  used,  the  instrument,  and  the  context  in  which  the  measurement  is  to  take 
place.  Often  the  value  of  a  particular  MoM  can  not  easily  be  observed  or  measured  and  one  or  more 
surrogate  measures  are  used  in  its  place.  In  any  event,  the  development  of  the  set  of  MoM  to  be  used  in  the 
study  anchors  the  process  that  will  eventually  lead  to  a  solution  strategy. 

This  process  (Figure  4.2)  revolves  around  the  conceptual  model  that  the  assessment  team  builds,  and  is  at 
the  heart  of  that  process.  It  is  best  practise  to  make  this  model  explicit  and  have  it  serve  as  the  common 
picture  that  develops  a  high  quality  of  shared  understanding  among  the  team,  sponsors,  stakeholders, 
and  other  key  study  participants.  The  initial  conceptual  model  consists  simply  of  the  MoM,  a  first  cut 
of  the  hypothesised  relationships  among  them,  assumptions  about  variables  and  their  relationships, 
and  constraints.  Later  iterations  include  additional  independent  variables  that  are  known  or  assumed  to 
affect  the  values  of  the  MoM  or  the  nature  of  the  relationship  among  them,  increasingly  detailed 
specifications  of  relationships,  and  specific  values  or  ranges  for  the  independent  variables. 

The  identification  of  human  and  organisational  factors  that  impact  model  variables  and  relationships 
serves  to  flesh  out  the  basic  conceptual  model  generated  in  the  problem  formulation  phase. 

Scenarios  then  need  to  be  derived  to  provide  opportunities  in  an  appropriate  context  for  data  collection  and 
exploration  of  the  variables  and  relationships  contained  in  the  conceptual  model.  The  data  the  study 
requires  are,  in  large  measure,  a  derivative  of  the  scenarios  utilised  and  the  design  of  the  assessment. 

The  design  of  an  assessment  also  requires  specification  of  methods  and  tools  and  how  they  will  be 
employed.  Methods  and  tools  are  required  to  explore  the  relationships  among  the  independent  variables 
and  between  the  independent  and  dependent  variables.  Complex  solution  strategies  may  be  necessary. 
In  these  cases,  multiple  analyses  will  be  implied.  The  problem  must  be  divided  into  parts,  each  part 
requiring  analysis  with  its  own  set  of  tools.  Frequently  the  tools  that  are  available  do  not  provide  interfaces 
from  one  part  of  the  analysis  to  the  next. 

Taken  together,  the  detailed  specification  of  the  MoM,  the  development  of  a  conceptual  model  including 
the  relevant  human  and  organisational  factors,  the  specification  of  a  set  of  scenarios,  and  a  data  collection 
and  analysis  plan  (that  consists  of  the  methods  and  tools  to  be  used)  constitute  a  solution  strategy. 

The  solution  strategy  developed  needs  to  be  tested  to  see  if  it  can  be  expected  to  address  the  issues  at  hand, 
within  schedule  and  resource  constraints,  and  with  accepted  levels  of  uncertainty  and  risk.  However, 
uncertainty  and  risks  are  being  continually  assessed  throughout  the  process  of  developing  a  solution 
strategy.  The  team  should  also  consider  the  form  of  study  output  and  its  relevance  to  the  decisionmaker. 
Iteration  of  these  ideas  with  the  stakeholders  throughout  the  study  helps  to  avoid  surprises  and  to  ensure 
that  the  basic  assumptions  underlying  the  study  have  not  changed. 


4,4  ITERATING  THE  STUDY  PLAN 

Figure  4.3  depicts  the  iterative  nature  of  the  process  involved  in  developing  the  Overall  Study  Plan, 
linking  problem  formulation  and  solution  strategies  together  with  the  inputs  from  study  sponsors  and  stake 
holders. 
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A  first  order  feedback  loop  is  shown  between  problem  formulation  and  solution  strategy,  with  both 
processes  having  iterative  internal  processes.  An  analysis  of  the  study  of  risk  and  uncertainty  provides  the 
control  mechanism  that  drives  the  iterative  process  to  an  acceptable  result. 


4.5  STUDY  MANAGEMENT  PLAN 

The  team  should  also  create  and  maintain  a  Study  Management  Plan  (SMP)  to  guide  the  direction, 
management  and  co-ordination  of  the  project  team.  The  SMP  should  include  a  detailed,  time-phased 
execution  plan  for  the  study  and  a  Work  Breakdown  Structure  (WBS)'.  The  SMP  should  show  the 
requirements  for  all  the  team  products  and  their  delivery  dates  thereby  creating  delivery  milestones  for  the 
execution  of  the  study.  It  should  show  the  planned  dates  for  all  scheduled  meetings  including  progress 
meetings  and  technical  interchange  meetings.  The  plan  should  also  include  a  time-phased  manning  plan 
identifying  the  types,  quantities,  and  period  of  performance  for  all  members  of  the  study  team.  The  SMP 
should  include  details  of  the  controls  that  will  be  applied  to  supervise  any  contractor  performance. 
The  team  should  maintain  a  current  version  of  the  SMP  during  the  study  period  of  performance.  The  SMP 
would  typically  include  the  associated  supporting  plans: 

•  Study  glossary; 

•  Analysis  plan; 

•  Tool  deployment  and  modelling  and  simulation  plan; 

•  Data  collection/engineering  plan; 

•  Configuration  management  plan; 


*  A  WBS  is  a  decomposition  of  the  effort  into  its  constituent  parts  (or  tasks)  and  the  assignment  of  assets  to  the  component 
tasks.  Assets  may  be  people’s  time,  facilities,  or  other  elements  required  to  complete  the  task.  WBS  are  common  in 
construction  and  engineering  projects. 
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•  Explicit  consideration  of  risk  and  uncertainty; 

•  Quality  assurance  plan; 

•  Security  plan; 

•  Review  plan;  and 

•  Plan  of  deliverables. 

The  elements  of  an  ideal  SMP  are  defined  and  discussed  below. 

4.5.1  Study  Glossary 

The  assessment  team  should  create  and  maintain  a  study  glossary  comprising  all  relevant  definitions 

needed  in  the  study.  It  should  aim  to  create  a  general  study  glossary  that  is  improved  by  every  study  that 

uses  this  glossary.  As  a  starting  point,  the  NATO  AAP-6,  “NATO  Glossary  of  Terms  and  Definitions”, 
or  the  Joint  Publication  1-02,  “DOD  Dictionary  of  Military  and  Associated  Terms”,  should  be  used. 

4.5.2  Analysis  Plan 

The  assessment  team  should  also  create  and  maintain  an  analysis  plan  for  the  study.  The  analysis  plan 
should  describe  the  analyses  in  detail.  This  description  should  include  the  analysis  methodology,  the  tools 
to  be  used  for  analysis,  the  input  data  requirements,  the  Essential  Elements  of  Analysis  (EEA),  the  MoM 
to  be  used  to  evaluate  the  results,  and  any  analysis  assumptions. 

4.5.3  Tool  Deployment  and  Modelling  and  Simulation  Plan 

The  team  may  create  and  maintain  a  tool  deployment  and  modelling  and  simulation  plan  covering  the 
needs  of  each  task  for  numerical  simulations  or  other  applied  means  of  operations  analysis  (OA).  It  should 
describe  the  use  of  tools  and  models  and  simulations  in  the  feasibility  study  (FS).  The  plan  should  include 
a  description  of  each  tool  to  be  used,  a  list  of  the  key  assumptions  and  caveats  for  each  tool,  an  analysis  of 
the  suitability  of  each  tool  in  addressing  the  functionality  and  performance  issues  in  the  FS,  the  source  of 
input  data  for  each  tool,  the  available  output  from  each  tool,  and  should  detail  any  changes  to  these  tools 
that  are  intended.  The  plan  should  indicate  how  data  traceability  from  one  tool  to  another  will  be 
maintained.  The  modelling  and  simulation  plan  may  be  included  as  part  of  the  analysis  plan  in  simple 
studies. 

4.5.4  Data  Collection/Engineering  Plan 

The  team  should  create  and  maintain  a  Data  Collection/Engineering  Plan  (DCEP)  which  covers  data  and 
metadata  necessary  to  describe: 

•  The  scenario; 

•  The  essential  elements  of  analysis  in  the  analysis  plan; 

•  The  MoM  to  be  used  to  evaluate  the  result  (also  in  the  analysis  plan);  and 

•  The  input  and  output  parameters  of  tools  to  be  used  within  the  study. 

The  DCEP  describes  and  documents  who  owns  the  data,  where  the  data  can  be  found  (including  open 
sources  like  the  Web),  necessary  methodologies  and  procedures  to  prepare  the  data,  and  assumptions  and 
constraints  connected  to  generated  data,  etc.  The  data  definitions  used  in  the  DCEP  have  to  be  harmonised 
with  the  study  glossary. 
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4.5.5  Configuration  Management  Plan 

The  team  should  create  and  maintain  a  configuration  management  plan.  The  plan  should  ensure 
identification,  traceability  and  control  of  the  descriptions  of  the  system  elements,  interfaces  and 
architectures  considered  in  the  FS,  as  well  as  associated  documentation.  The  plan  should  show  how 
the  study  database  must  be  controlled  and  updated.  The  plan  should  follow  NATO  STANAG  4159, 
“NATO  Materiel  Configuration  Management  Policy  and  Procedures  for  Multi-National  Joint  Projects” 
(1992),  as  a  guideline.  The  plan  should  also  ensure  that  the  description  of  the  system  configuration  being 
simulated  or  analysed  can  be  precisely  identified,  as  well  as  any  system  or  technology  improvements 
considered,  with  respect  to  an  identified  baseline  configuration. 

4.5.6  Study  Risk  Register 

The  team  should  also  identify  and  assess  the  technical  and  schedule  risks  concerned  with  the  successful 
completion  of  the  FS.  The  list  of  risks  identified  should  be  maintained  in  a  study  risk  register, 
which  shows  the  probability  of  occurrence  of  each  risk  and  its  impact  on  the  FS.  The  register  should 
include  the  risk  mitigation  activity  for  each  risk  and  the  expected  improvements  to  time  and  performance. 
The  study  risk  register  should  be  regularly  maintained,  should  be  available  to  the  assessment  team  by 
arrangement,  and  should  be  presented  at  each  progress  meeting.  The  Generic  Risk  Register  (GRR) 
developed  during  the  SAS-026  efforts  (Chapter  10)  is  recommended. 

4.5.7  Quality  Assurance  Plan 

The  team  should  create  and  maintain  a  quality  assurance  plan.  The  plan  should  declare  all  relevant  quality 
standards  and  procedures  that  are  to  be  applied  in  the  course  of  the  study,  and  should  describe  the  quality 
organisation  to  be  used,  including  the  principal  quality  officers  and  their  lines  of  authority.  The  policy  and 
requirements  for  quality  assurance  in  NATO  are  given  in  the  following  documents,  which  should  be  used 
as  guidelines: 

STANAG  4107,  “Mutual  Acceptance  of  Government  Quality  Assurance  and  Usage  of  the  Allied 
Quality  Assurance  Publications  (AQAPs)”,  6*  Edition  (March  1998). 

AQAP  100,  “General  Guidance  on  NATO  Quality  Assurance”,  2"*^  Edition  (March  1995) 
or  equivalent/comparable  national  standards. 

4.5.8  The  Security  Plan 

The  team  should  create  and  maintain  a  security  plan.  This  plan  should  contain  the  approach  to  the 
utilisation,  storage,  publication,  dissemination,  and  control  of  classified  and  unclassified  materials. 

4.5.9  Review  Plan 

The  team  should  create  and  maintain  a  review  plan.  For  every  critical  phase  of  the  study,  preferably 
marked  by  respective  milestones,  reviews  of  the  study  have  to  be  planned  and  executed.  Participants 
should  go  beyond  the  members  of  the  study  team  to  include  peer  reviews.  The  review  results  should  be 
documented. 

4.5.10  Plan  of  Deliverables 

The  team  should  also  create  and  maintain  a  plan  of  deliverables  for  each  phase  of  the  study.  This  includes 
what  is  to  be  delivered,  when  it  is  to  be  delivered,  to  whom,  in  what  form  and  format,  and  how  many 
copies. 
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4.6  CHAPTER  4  ACRONYMS 

C2  Command  and  Control 

DCEP  Data  Collection/Engineering  Plan 

EEA  Essential  Elements  of  Analysis 

FS  Feasibility  Study 

GRR  Generic  Risk  Register 

MoM  Measures  of  Merit 

OA  Operations  Analysis 

SMP  Study  Management  Plan 

WBS  Work  Breakdown  Structure 
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“It’s  best  to  know  what  you  are  looking  for,  before  you  look  for  it.  ”  —Winnie  the  Pooh,  from  A.  A.  Milne. 


In  order  to  understand  the  impaet  of  Command  and  Control  (C2),  it  is  neeessary  not  only  to  analyse  and 
measure  the  effeet  of  C2  on  military  operations,  but  also  the  effeets  on  the  eomponents  of  the  eonstituent 
systems.  No  single  measure  or  methodology  exists  that  satisfaetorily  assesses  the  overall  effeetiveness  of 
C2.  Therefore,  a  multi-faeeted  and  sometimes  multi-phased  approaeh  is  neeessary.  The  benefits  of  C2 
should  be  evaluated  through  their  impaet  on  the  fulfilment  of  the  military  and  poliey  objeetives,  and  the 
impaet  of  C2  should  be  measured  in  terms  of  speeifie  qualities  that  are  relevant  to  these  objeetives.  A  set 
of  seenarios  provides  the  eontexts  in  whieh  Measures  of  Merit  (MoM)  are  determined. 

MoM  Challenges 

During  the  last  two  deeades,  many  new  automated  C2  systems  have  been  developed  and  fielded.  However, 
the  determination  of  both  the  performanee  and  the  effeetiveness  of  these  systems  has  proven  to  be  a 
eomplex  problem.  Reeognising  this,  the  Military  Operations  Researeh  Soeiety  (MORS)  has  sponsored 
several  workshops  on  MoM  sinee  1985.  The  workshops  have  led  to  the  development  of  an  analysis 
framework.  Modular  Command  and  Control  Evaluation  Strueture  (MCES),  for  the  measurement  of 
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performance  and  effectiveness  within  a  conceptual  model  for  C2.  Based  on  the  MORS  workshops,  the  US 
Army’s  Training  &  Doctrine  Command  Analysis  Center  (TRAC)  developed  the  C2  Measures  of 
Effectiveness  (MoE)  Handbook  in  1990.  This  document  and  the  measurement  tools  developed  for  the 
Headquarters  Effectiveness  Assessment  System  (HEAT)  and  the  Army  Command  and  Control  Evaluation 
System  (ACCES)  represented  the  then  established  best  practices. 

The  AC/243  Panel  7  Ad  Hoc  Working  Group  (AHWG)  on  the  Impact  of  Command,  Control, 
Communications  &  Intelligence  (C3I)  on  the  Battlefield  acknowledged  that  the  specification  of  measures 
of  effectiveness  is  difficult.  The  1992  final  report  recommended  that  a  hierarchy  of  measures  be 
established  as  an  important  step  in  understanding  overall  system  effectiveness,  and  that  systems  be 
analysed  at  different  levels  of  detail.  The  types  of  measures  were  grouped  relating  to  C2  system 
performance,  force/commander  effectiveness,  and  battle  outcome.  To  quote  from  the  final  report, 
“Measures  ...  are  often  inadequate  and  too  model  or  scenario  specific.  In  addition,  they  have  often  been 
generated  in  ad  hoc  ways,  suggesting  a  lack  of  formal  analysis  in  their  development.”  Since  then,  RSG-19, 
SAS-002  and  SAS-026  have  canvassed  the  field,  and  have  brought  together  the  best  ideas  and  practices  in 
order  to  support  MoM  development  and  applications  within  C2  assessment'.  This  version  of  the  Code  of 
Best  Practise  (COBP)  extends  this  thinking  and  includes  the  Operations  Other  Than  War  (OOTW) 
domain. 

Definitions 

It  has  been  recognised  that  a  single  definition  for  measures  of  performance  (MoP)  and  effectiveness 
(MoE)  does  not  exist.  MoM  is  recommended  as  a  generic  term  to  encompass  different  classes  of  measures. 
The  measures  are  defined  in  hierarchical  levels  related  to  each  other,  each  in  terms  of  its  own  boundary. 
From  the  conceptual  viewpoint,  it  is  important  to  keep  in  mind  the  level  of  analysis  and  the  context  in 
which  the  measurements  are  made. 

Within  the  MCES  framework,  MORS  has  developed  a  four-level  hierarchy  of  measures  from  high-level 
force  effectiveness  to  low-level  rudimentary  measures  of  physical  entities,  which  were  adopted  by 
RSG-19.  In  the  context  of  OOTW,  a  fifth  level  is  added.  Measures  of  Policy  Effectiveness  (MoPE), 
to  characterise  the  contribution  of  military  actions  to  broader  policy  and  societal  outcomes.  For  OOTW, 
political  factors  are  paramount  and  considerations  such  as  media  coverage,  local  regional  stability, 
and  sustainment  of  community  societal  standards  must  be  taken  into  account.  Military  missions  may  not 
directly  achieve  policy  objectives,  although  they  often  strive  to  provide  an  environment  more  conducive  to 
these  objectives.  However,  MoE  of  military  tasks  should  quantify  performance  against  military  missions, 
not  the  overall  political  aspirations. 

The  Code  of  Best  Practice  has  adopted  the  following  five  levels  of  MoM: 

•  Measures  of  Policy  Effectiveness  (MoPE),  which  focus  on  policy  and  societal  outcomes; 

•  Measures  of  Force  Effectiveness  (MoFE),  which  focus  on  how  a  force  performs  its  mission  or  the 
degree  to  which  it  meets  its  objectives; 

•  Measures  of  C2  Effectiveness  (MoCE),  which  focus  on  the  impact  of  C2  systems  within  the 
operational  context; 

•  Measures  of  Performance  (MoP),  which  focus  on  internal  system  structure,  characteristics  and 
behaviour;  and 

•  Dimensional  Parameters  (DP),  which  focus  on  the  properties  or  characteristics  inherent  in  the 
physical  C2  systems. 


'  The  following  reports  can  be  ordered  from  http://www.rta.nato.int,  RTO-MP-029  AC/323(SAS)TP/9.  Workshop  on  C3I 
Modelling  (April  1999),  RTO-MP-038  AC/323  (SAS)  TP/12  Modelling  and  Analysis  of  Command  and  Control  (June  1999), 
RTO-TR-9  AC/323  (SAS)  TP/4  COBP  on  Command  and  Control  (March  1999). 
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Figure  5.1  emphasises  the  diminishing  impact  of  a  particular  MoM  as  the  circle  widens. 


Environment 


5.1  MEASUREMENT  OBJECTIVES 

The  important  issues  raised  by  decisionmakers  require  a  sense  of  the  degree  to  which  C2  performance  may 
improve  force  and  policy  effectiveness.  Therefore,  C2  assessments  are  often  called  upon  to  provide 
convincing  evidence  of  the  expected  improvements  in  mission  effectiveness  that  can  be  attributed  to 
improved  C2.  The  ideal  approach  may  be  to  define  a  single  measure  that  reflects  the  military  and  political 
objectives  of  the  missions  under  consideration.  Flowever,  the  determination  of  such  a  measure  is  generally 
not  feasible,  although  not  necessarily  impossible  for  particular  classes.  In  most  decisionmaking  problems, 
it  is  necessary  to  define  several  measures  that  together  provide  the  necessary  insights.  A  major  reason  for 
this  is  that  a  single  measure  may  not  provide  sufficient  scope  and/or  detail  to  analyse  the  impact 
of  specific  C2  elements,  particularly  second  and  third  order  effects  or  unintended  consequences. 
Many  analyses  are  conducted  precisely  in  order  to  enable  trade-off  between  important  equities  which  can 
only  be  seen  if  a  set  of  MoM  is  generated  for  analysis.  The  set  of  MoM  selected  must  be  comprehensive  to 
ensure  that  all  factors  are  considered. 

MoM  are  used  to  compare  different  options  on  equal  terms,  and  serve  a  wide  range  of  purposes,  including: 

•  Establishing  a  standard  or  expectation  of  performance  (for  new  requirements); 

•  Establishing  the  bounds  of  performance  of  a  system  as  well  as  the  effects  of  imposed  constraints; 

•  Comparing  and  selecting  alternative  systems  that  may  be  very  dissimilar  but  are  designed  to 
achieve  a  similar  purpose; 

•  Assessing  the  utilisation  of  a  system  in  new  or  unexpected  application  domains  or  missions; 
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•  Identifying  potential  weaknesses  in  specific  areas  of  an  organisation  or  system  (areas  of  high  error 
potential  or  high  user  workload); 

•  Analysing  the  impacts  of  organisational  changes; 

•  Analysing  training  effectiveness; 

•  Determining  the  most  cost-effective  approaches  to  achieve  desired  objectives; 

•  Comparing  a  replacement  system,  or  components  of  a  system,  against  predecessors  or 
competitors; 

•  Assisting  in  generating  and  validating  requirements  and  deriving  specific  C2  requirements  from 
broad  statements  of  objectives; 

•  Evaluating  the  effectiveness  of  human  decisionmaking  in  the  C2  cycle; 

•  Determining  the  degree  of  mission  success;  and 

•  Determining  the  return  to  normality  in  OOTW. 


5.2  RELATIONSHIPS  AMONG  MoM 

The  development  of  an  operational  definition  of  the  measure,  the  development  of  instruments, 
the  application  of  the  instruments  to  collect  of  appropriate  data,  and  the  establishment  of  relationships 
among  a  group  of  MoM  vary  in  difficulty,  effort,  cost,  precision,  generalisability  and  other  characteristics. 
These  differences  are  often  related  to  the  type(s)  of  MoM  involved  and  the  domain  in  which  the 
measurement  needs  to  take  place. 

C2  Assessments  involve  measurement  of  variables  that  exist  in  the  physical,  information,  and  cognitive 
domains^.  In  general,  the  development  of  operational  definitions,  instrumentation  and  data  collection  for 
variables  to  be  measured  in  the  physical  and  information  domains  are  more  straight  forward  and  require 
less  effort  and  expense  than  dealing  with  variables  that  are  measured  in  the  cognitive  domain. 
Furthermore,  the  former  can  be  measured  more  “precisely”,  are  easier  to  comprehend,  and  are  less  subject 
to  interpretation  than  the  latter. 

In  general,  DP,  MoP  related  to  systems,  and  MoFE  tend  to  be  measured  in  the  physical  domain 
(e.g.  bandwidth,  computing  capacities,  time  to  accomplish  a  task,  force  exchange  ratios)  while  MoP 
related  to  measures  of  C2  effectiveness  (quality  of  awareness,  shared  awareness,  and  trust)  and  MoPF 
(will  of  an  adversary,  public  opinion)  tend  to  be  measured  in  the  cognitive  domain. 

As  one  goes  up  the  hierarchy  of  MoM,  the  measures  tend  to  become  more  context,  task,  or  mission 
specific.  For  example,  the  performance  characteristics  of  systems  (DP)  apply  to  systems  in  general  but 
MoFE  are  usually  limited  to  a  set  of  tasks  or  missions.  MoFE  for  combat  are  very  different  from  MoFE  for 
various  OOTW.  If  done  well  measures  of  C2  effectiveness  will  be  scenario  independent  so  one  can 
compare  C2  effectiveness  across  a  range  of  missions  and  circumstances. 

Except  for  DP  any  of  the  MoM  can  be  either  an  independent  or  dependant  variable  in  a  given  assessment 
with  any  of  the  independent  variables  being  either  “controllable”  or  not.  The  difficulty  in  establishing 
relationships  among  the  MoM  varies  as  a  function  of  the  level  of  the  independent  variable. 

The  Assessment  Team  should  recognise  and  plan  for  the  difficulties  associated  with  using  various  MoM 
and  should  avoid  substituting  easier  to  deal  with  but  less  relevant  MoM.  It  is  always  better  to  try  to 


2 

David  S.  Alberts,  John  J.  Garstka,  Richard  E.  Hayes,  and  David  A.  Signori,  Understanding  Information  Age  Warfare 
(Washington,  DC:  CCRP  Publication  Series,  August  2001). 
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measure  (estimate,  approximate)  MoM  that  reflect  first  order  effects  than  to  precisely  measure  MoM  that 
do  not  adequately  reflect  key  aspects  of  the  problem. 


5.3  MEASURING  MoM 

This  section  outlines  measurement  theory  concepts  that  apply  to  ensure  that  the  right  measuring 
instruments  are  selected  and  applied  correctly.  By  definition,  measurement  is  the  assignment  of  values  to 
observation  units  that  express  properties  of  the  units.  Four  levels  of  measures  relate  numbers  to  properties 
of  interest:  nominal  (e.g.  artillery  vs.  infantry),  rank  or  ordinal  (e.g.  worst  to  best),  relative  or  interval 
value  (e.g.  change  in  temperature),  and  absolute  value  or  ratio  (e.g.  2  kilobits  per  second).  Analysts  must 
ensure  that  the  specific  MoM  adopted  are  at  the  appropriate  levels  of  measurement. 

The  key  properties  for  quality  assurance  are  reliability  and  validity.  Other  significant  properties  include 
practical  issues,  such  as  the  effort  required  to  collect  appropriate  data  and  the  convenience  of  measurement 
(e.g.  whether  the  collection  process  itself  interferes  with  the  conduct  of  an  exercise  or  experiment). 
Ideally,  measurements  should  be  easy  to  capture  and  easy  to  apply.  There  are  clearly  trade-offs  to  be  made 
between  MoM  that  may  closely  track  the  property  of  interest  and  that  are  costly  and/or  difficult  to  measure 
and  those  that  are  less  strongly  related  to  the  property  of  interest  but  that  are  easier  to  measure.  The  effort 
required  for  collection  bears  no  direct  relationship  with  validity,  but  reliability  may  be  related  to  cost. 
Reliable  measurements  require  repeated  observations  and  appropriate  sample  sizes.  Reliability  represents 
accuracy  and  consistency.  A  cost-effective  measurement  plan  provides  enough  data  for  useful  and 
definitive  conclusions.  However,  cost  and/or  convenience  of  measurement  may  be  an  overriding  factor  in 
system  evaluation. 

Failure  to  take  validity  and  reliability  into  account  raises  the  risk  of  generating  false  conclusions.  Validity 
and  reliability  are  not  absolutes,  but  matters  of  degree.  Validity  is  the  degree  to  which  a  measure 
characterises  the  attribute  of  interest  and  only  that  attribute.  Complex  concepts  often  require  multiple 
measures  to  provide  valid  information.  In  order  to  make  a  valid  link  between  the  performance  of  a  system 
as  a  whole  against  performance  of  its  components,  the  measures  must  correspond  to  critical  tasks. 
Reliability  represents  accuracy  and  repeatability.  A  measure  may  be  reliable  but  not  valid,  or  it  may  be 
valid  but  not  reliable. 

5,3,1  Validity 

The  properties  of  validity  may  be  categorised  into  five  types:  internal,  construct,  statistical,  external, 
and  expert: 

•  Internal  validity  is  defined  as  the  establishment  of  causal  relationships  between  variables  of 
interest.  This  is  necessary  to  accept  a  hypothesis  that  a  given  measure  is  responsible  for  a  specific 
effect  on  another  measure; 

•  Construct  (also  referred  to  as  content)  validity  means  that  the  target  objects,  and  only  the  target 
objects,  are  measured; 

•  Statistical  validity  implies  that  sufficient  sensitivity  is  involved  in  order  to  determine  relationships 
between  independent  and  dependent  variables.  Statistical  tests  control  two  types  of  errors  in 
measurement.  Type  1,  or  alpha,  is  the  probability  of  rejecting  a  claimed  hypothesis  that  is  true. 
Type  If,  or  beta,  is  the  probability  of  accepting  an  hypothesis  that  is  not  true; 

•  External  validity  implies  that  the  results  may  be  extended  to  other  populations  or  environments; 
and 

•  Expert  validity  refers  to  the  degree  to  which  measures  are  accepted  by  those  knowledgeable  in  the 
field. 
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A  MoM  should  meet  the  validity -related  eriteria  outlined  in  Table  5.1. 


Table  5.1:  Validity  Criteria  of  Measures 


Validity  Criterion 

Definition 

Mission  Oriented 

Relates  to  force/system  mission 

Realistic 

Relates  realistically  to  the  C2  system  and  associated  uncertainties 

Appropriate 

Relates  to  acceptable  standards  and  analysis  objectives 

Inclusive 

Reflects  those  standards  required  by  the  analysis  objectives 

Discriminatory 

Able  to  distinguish  measurement  parameters 

Meaningful 

Accepted  by  subject  matter  experts 

Simple 

Easily  understood  by  users 

Relevant 

Target  objects,  and  only  the  target  objects,  are  measured 

Generalisable 

Extent  to  which  results  may  be  extended  to  other  populations  or 
environments 

5,3,2  Reliability 

Reliability  involves  the  expeetation  of  errors  assoeiated  with  measurements.  It  is  defined  as  the  aeeuraey 
of  a  measurement,  as  refieeted  in  the  varianee  of  repeated  measurements  of  the  same  phenomenon. 
The  key  prineiples  of  reliability  are  eonsisteney  (repeatability)  and  aeeuraey.  The  varianee  assoeiated  with 
measurement  must  be  known  or  estimated  to  interpret  results  and  to  diseriminate  between  real  effects  and 
measurement  effects. 

The  reliability-related  criteria  that  MoM  should  meet  are  outlined  in  Table  5.2.  Additional  criteria  that 
have  proven  usual  in  the  past  are  outlined  in  Table  5.3. 


Table  5.2:  Reliability  Criteria  of  Measures 


Reliability  Criterion 

Definition 

Discriminatory 

Identifies  real  differences  between  alternatives 

Measurable/Observable 

Available  for  measurement 

Quantitative 

Can  be  assigned  values,  numbers,  or  ranks 

Objective 

Defined  or  derived,  independent  of  subjective  opinion 

Sensitive 

Able  to  ascertain  changes  in  system  variables 

Consistent 

Same  results  are  obtained  on  repetition 
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Table  5.3:  Other  Useful  Criteria 


Criterion 

Definition 

Timeliness 

Adaptability 

Flexibility 

Robustness 

Sampling  rate  adequate  to  detect  changes 

Sufficient  within  available  resources 

Capability  across  a  range  of  anticipated  situations 

Capability  across  a  range  of  unanticipated  situations 

5.4  PRACTICAL  MoM  ISSUES 

The  assessment  of  C2  requires  the  application  of  a  framework  to  yield  values  for  appropriate  MoM. 
Analyses  of  C2  systems  and  processes  often  reveal  a  complex  hierarchical  composition.  A  structured 
resolution/functional  decomposition  approach  may  be  related  to  the  organisational  structure  to  yield 
performance  measures  for  the  organisation  as  a  whole,  individual  components  within  the  organisation, 
and  specific  tasks  within  the  organisational  cells. 

If  the  analyst  assumes  that  C2  effectiveness  is  positively  correlated  with  overall  military  unit  effectiveness, 
MoM  could  be  obtained  by  addressing  the  outcomes  or  products  of  such  unit  activities.  Goal-level 
evaluation  attempts  to  define  the  ability  of  the  specific  military  formation  to  make  the  system  state  match 
the  goal  (directive)  provided  by  the  superior  headquarters.  These  are  measures  of  force  effectiveness. 
The  degree  to  which  the  system  state  matches  the  desired  goal  states  indicates  a  level  of  effectiveness. 
Alternatively,  C2  effectiveness  may  be  viewed  as  dependent  on  the  functional  processes  of  the  C2  system, 
with  measures  obtained  mainly  at  the  task  level. 

A  C2  assessment  framework  encompasses  several  factors  that  must  be  considered  iteratively,  as  discussed 
in  the  introductory  chapter.  Typical  factors  important  for  the  identification  and  selection  of  MoM  include 
the: 

•  Assessment  configuration,  e.g.,  storyboard,  testbed,  constructive  simulation,  field  trial; 

•  Assessment  goal  or  purpose; 

•  Context,  assumptions,  and  constraints; 

•  Scenarios  or  stimuli; 

•  Collection  means,  e.g.,  subject  matter  experts,  automatic  data  logging; 

•  Analysis  plan;  and 

•  Interpretation  of  results. 

5,4,1  Categories  of  Measures 

A  common  thread  in  the  approaches  for  C2  assessment  is  the  functional  decomposition  of  the  C2  cycle. 
C2  effectiveness  depends  upon  the  functional  processes  of  the  C2  system,  and  the  evaluation  of  functions 
may  be  determined  by  data  measured  at  the  task  level. 

The  evaluation  of  tasks  provides  the  most  detailed  insight  into  C2  activities.  The  primary  measures  are 
expressed  in  terms  of  time  consumed  and  accuracy.  Task  analysis  must  be  performed  prior  to  evaluation, 
with  the  identification  of  task  definition  and  the  critical  elements  for  successful  task  completion. 
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Measures  of  a  C2  system’s  behaviour  may  thus  be  reduced  to  measures  based  on  time,  accuracy,  or  a 
combination  which  may  be  interdependent.  Time  based  measures  are  quantitative,  while  accuracy 
measures  may  be  quantitative  or  qualitative. 

For  C2  tasks,  time-based  metrics  include  the: 

•  Time  taken  to  react  to  an  event  (time  to  notice  process  and  act  upon  new  information); 

•  Time  to  perform  a  task  (time  to  make  decision); 

•  Time  horizon  for  future  for  predictive  analysis;  and 

•  Rate  of  performing  tasks  (tempo). 

Metrics  for  accuracy  include: 

•  Precision  of  the  observed  system(s)  performance; 

•  Reliability  of  the  observed  system(s)  performance; 

•  Completeness  (known  unknowns,  unknown  unknowns); 

•  Errors  (alpha,  beta,  omission,  transposition,  severity);  and 

•  Quality  of  information  produced. 

Some  accuracy  measures  may  be  calculated  in  units  of  time,  e.g.,  the  time  taken  to  detect  an  error. 
Quality  of  decisions  is  difficult  to  evaluate  objectively,  except  by  focusing  on  outcomes.  The  processes 
involved  may  have  to  be  examined  to  obtain  objective  measures,  or  subject  matter  experts  may  be 
consulted  to  make  an  evaluation.  Accuracy  of  information  implies  both  the  accuracy  of  the  data  and  the 
accuracy  of  the  interpretation  of  the  data. 

Time  based  and  accuracy  measures  often  bear  an  inverse  relationship,  implying  a  trade-off  between  speed 
of  performance  and  accuracy  of  performance.  Speed  of  performance  must  be  specified  in  terms  of 
minimum  desired  accuracy  or  completeness,  and  accuracy  measurements  in  terms  of  time  available. 
Therefore,  the  specification  of  thresholds  or  standards  for  metrics  must  be  referenced  in  terms  of  imposed 
constraints. 

Examples  of  time  and  accuracy  based  measures  are  compiled  in  Table  5.4.  Table  5.5  provides  some 
additional  examples,  specifically  MoP  and  MoCE. 


Table  5.4:  Examples  of  Time  and  Accuracy-Based  Measures 


Measure 

Example 

Time-Based 

Time  to  perform  fixed  or  sequence  of  tasks 

Planning  tasks 

Time  to  perform  a  variable  task 

Developing  and  selecting  options  or  courses  of  action 

Time  to  recognise  or  respond  to  an  event 

Response  to  a  critical  enemy  contact 

Time  to  achieve  a  target  state 

Tactical  objective 

Percentage  of  time  on  target 

Data  bases  up  to  date 

Number  of  events  in  queue 

Messages  pending  action 

Timeliness  of  responses 

Fire  plan  schedule 
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Table  5.4:  Examples  of  Time  and  Accuracy-Based  Measures  (cont’d) 


Measure 

Example 

Accuracy-Based 

Accuracy  or  precision  of  performance  of  tasks 

Information  on  maps,  data  bases 

Sensitivity  of  detecting  system  events 

Recognition  of  events  requiring  change  in  plans 

Probability  of  making  errors 

Errors  in  fire  plan  target  schedules 

Time  to  recognise  existence  of  error 

Necessity  for  plan  alteration 

Time  to  recover  from  error 

Time  to  redo  part  of  plan 

Knowledge  of  current  system  status 

Comprehension  of  battle  situation 

Quality  of  decisionmaking 

Quality  of  tactical  plan 

Table  5.5:  Examples  of  MoPs  and  MoCEs 


MoPs  Technical  Services  Attributes  -  Hardware  and  Software 

Availability 

Functional  capabilities  available  to  users 

Survivability 

Ability  to  survive  partial  destruction  of  system 

Robustness/Endurance 

Ability  to  adapt  to  environment 

Maintainability 

Ease  of  repair  or  replacement  during  operation 

Computation  Capacity 

Acceptable  response  times  to  users 

Portability 

Ability  to  operate  on  different  platforms 

Mobility 

Ability  to  move  with  operational  units 

MoPs  Technical  Services  -  Applications  Attributes 

Interoperability 

Communications  with  other  C2  systems 

Security 

Confidentiality  and  integrity  of  data 

Confidentiality 

Information  protected  at  appropriate  level 

Integrity 

Required  for  confidence  of  data 

Customisability 

Ability  to  customise  parameters  to  actual  activities 

Quantity  of  Information 

Provide  all  information  required  by  user 

Bandwidth 

Ability  to  support  multi-media 

MoCEs  User  Effectiveness  -  Information  Quality 

Selectivity 

Ability  to  provide  required  information  in  required  amount 

Accuracy 

The  extent  to  which  true  values  are  approached 

Comprehension 

Facilitate  understanding  of  situation 
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Table  5.5:  Examples  of  MoPs  and  MoCEs  (cont’d) 


MoCEs  User  Effectiveness  -  Time  Related 

Response  time 

Response  to  requests  within  established  times 

Timeliness 

Information  available  at  appropriate  time 

Ease  of  use 

Ease  of  access  to  information 

Training  time 

Time  to  train  users 

Decision  response  time 

Time  available  to  commanders 

5.4.2  Example  Headquarters  C2  MoM 

C2  measures  may  also  be  divided  into  sets  corresponding  to  the  sequential  steps  of  the  C2  cycle. 
These  include: 

•  Monitoring  and  understanding:  information  transmission,  values,  times,  effect,  comprehension; 

•  Planning:  information  exchange,  co-ordination,  impact,  flexibility,  process  quality;  and 

•  Directing  and  disseminating. 

The  MoM  for  C2  can  also  be  focused  on  four  levels:  a  network  of  headquarters,  a  single  headquarters, 
the  individual  cells  within  the  headquarters,  and  performance  of  specific  tasks  within  the  cells. 

5.4.3  OOTWMoM 

While  national  NATO  policies  require  that  military  forces  be  prepared  for  high  intensity  conflict,  forces 
have  been  increasingly  involved  in  low-intensity  conflicts  and  C2  analyses  for  OOTW  are  therefore 
becoming  important.  OOTW  include  force  deployment  to  create  or  maintain  conditions  for  a  political 
solution  in  order  to  avoid  escalation  into  hostilities.  Threats  to  international  and  national  security  may  also 
unfold  from  natural  disasters,  terrorism  organised  crime,  civil  unrest,  migration,  or  other  territorial 
intrusions.  Most  OOTW  are  inherently  joint  or  combined  operations. 

While  the  determination  of  MoM  has  been  stated  as  difficult  to  obtain,  OOTW  offer  even  greater 
challenge  for  MoM.  Traditional  MoFE  such  as  loss  exchange  ratios,  combat  effectiveness,  or  duration  of 
the  campaign  are  rarely  applicable  to  OOTW.  In  such  operations,  military  forces  may  play  important  roles 
but  political  concerns  may  limit  the  scope  of  imposable  solutions.  Public  and  political  pressures  may  result 
in  shifts  in  the  selection  of  criteria  for  MoM  e.g.,  more  emphasis  may  be  placed  on  personnel  casualties 
and  less  on  equipment  losses. 

5.4.4  OOTWMoPE 

While  MoFE  and  MoCE  provide  measures  of  success  for  military  operations,  MoPE  measure  the  degree 
of  attaining  political  objectives.  In  some  cases,  such  as  humanitarian  assistance  or  nation  building  MoPE 
may  measure  the  degree  of  improvement  in  the  quality  of  life  of  the  populace. 

MoFE  usually  were  the  highest  MoM  used  within  the  analysis  of  Article  V  missions^  assuming  that 
effectiveness  is  directly  related  to  the  higher  level  MOPE  such  as  “winning  the  war”.  However,  such  an 
assumption  may  not  always  apply  to  OOTW.  For  example,  military  actions  that  would  be  highly  effective 
in  accomplishing  mission  objectives  in  war  might  be  quite  counterproductive  in  OOTW.  In  fact,  the  value 


^  Article  V  of  the  North  Atlantic  Treaty,  which  deals  with  classic  military  attacks  on  members. 
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of  military  actions  in  OOTW  is  not  so  much  a  question  of  physical  effects,  but  rather  how  military  actions 
and  their  physical  effects  are  perceived  by  the  various  actors  and  the  population  in  the  theatre,  how  the 
military  actors  interpret  the  behaviour  of  the  other  actors  and  the  critical  mission  task  conditions  such  as, 
for  example,  political  interest  and  media  attention. 

McCafferty  and  Lea  developed  low-level  military-related  measures  (MoCE)  to  cover  OOTW  (McCafferty 
and  Lea,  1997).  The  MoM,  which  are  classified  as  MoCE,  include  the: 

•  Time  between  the  arrival  of  friendly  forces  in  the  area  and  their  deployment; 

•  Time  between  deployment  of  friendly  forces  and  contact  with  adversary  forces; 

•  Length  of  time  adversary  forces  were  under  observation  without  posing  a  threat  to  friendly  forces; 

•  Length  of  time  friendly  forces  are  in  potential  danger  (i.e.  adversary  forces  have  the  opportunity 
to  fire  on  friendly  forces);  and 

•  Time  horizon  of  friendly  C2  processes  (how  far  into  the  future  they  are  focused). 

Mobility  may  be  important  for  OOTW,  as  well  as  sustainability  and  self-sufficiency  in  theatre,  with  the 
implication  of  emphasising  measures  of  reliability  and  maintainability.  Moreover,  the  perception  of  the 
capabilities  of  deployed  forces  acts  as  deterrence  or  coercion  on  the  parties  in  conflict. 

Some  examples  identified  by  McCafferty  and  Lea  are: 

•  Opportunities  to  employ  forces,  which  reflects  the  range  of  military  capabilities  available; 

•  Strategic  deployment,  which  is  related  to  deploying  and  recovering  the  right  force  to  theatre 
efficiently  and  in  time; 

•  Endurance,  to  maintain  an  effective  force  in  theatre  for  an  extended  time; 

•  Mission  objectives,  to  measure  the  success  of  achieving  military  objectives  in  OOTW;  and 

•  Successful  termination,  to  deal  with  progress  to  the  desired  end  state  (the  criteria  may  be  political 
and  thus  not  measured  by  military  activities). 

One  class  of  effectiveness  indicators  in  OOTW  is  provided  by  transition  measures,  which  focus  on  the 
progress  in  the  transfer  of  responsibilities  to  the  follow-on  military  force  or  civil  agency.  Transition 
measures  focus  on  the  degree  that  follow-on  organisations  assume  tasks  and  responsibilities. 

Progress  toward  success  may  be  tracked  by  normality  indicators,  which  are  indirect  measures  of 
the  effects  of  military  involvement  in  OOTW,  although  causal  relationships  are  difficult  to  prove 
(Lambert,  2000).  These  MoPE  may  be  obtained  by  evaluating  the  extent  to  which  conditions  have  been 
restored. 

Normality  indicators  measure  the  level  of  improvement  in  the  quality  of  life  of  the  general  population, 
and  may  be  defined  as  “relative  measures  of  the  state  of  normalcy  characterising  an  element  of  the  civil 
environment,  through  data  collected  on  a  regular  basis  and  assessed  to  have  the  frequency,  quantity, 
consistency  and  coverage  required  to  make  a  useful  objective  assessment  of  the  changes  occurring  in  the 
civilian  populace.”  (Department  of  National  Defence,  Canada,  1999).  Normality  indicators  can  be  grouped 
in  categories  and  adapted  to  meet  the  changing  requirements:  political,  socio-economic  levels  of 
development,  cultural,  legal  and  technological. 
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Table  5.6:  Normality  Indicators 


Criterion 

Examples 

Political 

Elections,  political  participation 

Economic 

Unemployment,  interest  rates,  and  market  baskets 

Social 

Number  of  students  in  schools,  number  of  refugees 

Technological 

Telephone  system  availability 

Legal 

Judicial  system  functioning 

Environmental 

Roads,  water  supply,  power  supply 

Cultural 

Sports  events,  concerts 

Limitations  of  normality  indicators  include: 

•  Inexperienced  personnel  available  for  data  collection  and  analysis.  A  mix  of  inexperienced  civil 
and  military  personnel  are  often  assigned  to  collect  data  in  fields  foreign  to  them;  training  may  be 
necessary  to  assure  reliable  and  valid  data; 

•  Temporary  effect  due  to  military  presence  for  data  collection.  The  mere  presence  of  military 
personnel  collecting  data  may  affect  normality  measurements; 

•  Difficulties  in  obtaining  valid  and  reliable  data  calibrated  against  baselines.  It  may  be  difficult  to 
establish  the  threshold  for  “normality”  if  archive  data  is  not  available; 

•  Extrapolation  in  space  and  time  from  a  specific  locality.  It  may  be  inappropriate  to  extrapolate 
civil  progress  to  an  entire  region.  Sampling  is  important; 

•  Limited  resources  and  constraints  for  collection  and  analysis.  Data  must  be  collected  consistently 
but  may  be  occasionally  unobtainable  due  to  physical  inaccessibility  or  lack  of  personnel; 

•  A  “snapshot”  which  may  not  provide  trends  if  infrequently  obtained.  Trend  analysis  requires 
sufficient  data  and  sampling  rates;  and 

•  During  OOTWs,  the  relevant  MoM  may  change  over  time,  particularly  the  lower  level  MoCE  and 
MoP.  For  example,  during  the  earliest  phases  of  NATO  operations  in  Bosnia,  tracking  weapon 
systems  and  knowing  how  many  of  them  were  in  cantonments  was  a  major  MoM.  Once  the  forces 
were  separated  and  most  weapon  systems  under  control,  emphasis  shifted  to  the  disruptive 
activities  such  as  road  blocks  and  ethnic  harassment.  As  these  behaviours  became  less  frequent, 
NATO’s  emphasis  shifted  to  normality  indicators. 

5,4,5  MoM  Hierarchies:  Some  Examples 
Modular  Command  and  Control  Evaluation  System 

Evaluation  of  C2  effectiveness  requires  a  comprehensive  approach  for  the  preparation  of  the  evaluation 
process,  the  collection  of  data,  and  its  interpretation.  MCES  addresses  both  the  managerial  and  analytical 
aspects  of  evaluation  and  was  originally  developed  for  the  systematic  comparison  of  C2  systems. 
The  objective  of  MCES  is  to  guide  analysts  in  the  identification  of  appropriate  measures  for  estimating  the 
effects  of  C2  on  combat. 

MCES  prescribes  a  process  of  measurement,  but  does  not  identify  either  a  measurement  system  or  a  set  of 
measures.  Similarly,  while  calling  for  the  collection  of  data,  MCES  does  not  provide  details  on  how  data 
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are  to  be  colleeted.  MCES  does  provide  guidanee  on  how  good  measures  and  good  eolleetion  proeedures 
are  eharaeterised,  but  leave  the  details  of  the  measurement,  data  eolleetion,  and  analysis  plans  to  the 
analyst. 

MCES  eonsiders  C2  as  eonsisting  of  three  eomponents: 

•  Physieal  entities  (equipment,  software,  people); 

•  Strueture  (interrelationships  of  entities);  and 

•  Processes  (C2  functions). 

The  boundary  of  a  C2  system  here  is  defined  as  a  delineation  between  the  system  studied  and  the 
environment.  The  US  Army’s  Training  and  Doctrine  Command  (TRADOC)  C2  MoCE  Handbook  adds 
mission  objective  as  the  top  layer  of  the  hierarchy  of  C2  components. 

MCES  focuses  on  measures  as  opposed  to  models,  but  includes  the  cybernetic  loop  model  of  generic  C2. 
It  consists  of  seven  procedural  steps. 

HEAT 

While  originally  developed  for  theatre-level  combat  applications,  the  HEAT  system  has  proven  robust. 
For  example,  it  has  been  used  to  assess  US  Department  of  State  crisis  task  force  performance, 
military  operations  in  Grenada,  Panama,  and  Haiti,  and  several  exercises  focused  on  peacekeeping 
and  humanitarian  assistance  missions.  The  underlying  C2  process  of  monitoring,  understanding, 
and  developing  alternative  actions;  predicting  the  consequences  of  each  alternative  for  each  possible  future 
under  consideration,  decision;  developing  and  promulgation  of  plans  and  directives  and  requests  for 
support;  seeking  information;  making  reports;  and  responding  to  inquiries  are  all  relevant  across  the 
ranges  of  OOTW  missions.  HEAT  has  been  modified  since  2000  to  include  measure  of  the  quality  of 
collaboration,  which  is  often  a  key  process  in  OOTW. 

ACCES 

ACCES  is  a  derivation  of  HEAT,  which  was  developed  primarily  for  joint  theatre-level  operations. 
ACCES  reorganised  HEAT  concepts  into  army  doctrinal  language  and  doctrine,  but  shares  the  same 
philosophy.  ACCES  has  been  applied  to  numerous  division  and  corps  command  centre  assessments. 
It  represents  a  comprehensive  set  of  practical  and  objective  performance  measures  for  C2  activities. 
The  primary  focus  of  ACCES  is  the  overall  performance  of  a  command  centre  or  network  of  command 
centres,  at  various  stages  of  the  C2  process,  from  the  collection  of  data  to  the  conversion  of  data  to 
intelligence  to  the  implementation  of  plans  and  directives.  The  underlying  approach  to  ACCES  is  that  C2 
comprises  interdependent  sub-processes  which  can  be  observed  and  measured.  ACCES  considers  C2  as  an 
adaptive  control  process,  where  information  collected  from  the  outside  is  processed  internally  to  generate 
plans  that  may  be  adapted  to  reflect  new  information.  ACCES  takes  the  view  that  the  overall  effectiveness 
of  a  command  centre  can  be  judged  by  the  viability  of  its  plans.  A  good  plan  is  one  that  can  be  executed 
without  the  need  for  modification  beyond  the  contingencies  stated  in  the  plan  and  that  remains  in  effect 
throughout  its  intended  life. 

Multi-Attribute  Utility  Theory  (MAUT) 

MAUT  is  similar  to  ACCES  in  the  sense  that  both  use  functional  decomposition  and  function-specific 
evaluation  metrics.  The  major  differences  are  that  MAUT  can  be  used  with  any  set  of  metrics 
(including  those  from  ACCES),  which  must  be  specified  by  the  analyst.  MAUT  assigns  weights  to  the 
MoM  at  each  level  of  the  MoM  hierarchy  and  utility  values  or  scores  at  the  lowest  level.  MAUT  then 
aggregates  upwards  the  weighted  scores  to  provide  composite  scores  of  effectiveness.  MAUT,  if  properly 
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used  with  appropriate  application  of  judgmental  weights,  will  allow  integrated  analyses  based  on 
multiple  MoM.  While  this  is  often  satisfying  to  decisionmakers  (it  provides  a  single  index  of  quality), 
analysis  should  always  monitor  the  components  of  such  indices  as  they  may  provide  insight  into  strength 
and  weaknesses  of  the  C2  system.  For  example,  many  OOTW  C2  problems  involve  a  variety  of  objective 
functions  and  trade-offs.  MAUT  results  should  always  be  assessed  by  sensitivity  analyses. 

Many  applications  of  MAUT  assume  additive  composition  of  MoM.  Flowever,  this  is  a  very  restrictive 
assumption  that  needs  to  be  validated  in  each  case.  In  addition  MoM  may  interact  suggesting  that  the 
aggregation  of  MoM  is  at  partly  multiplicative  (Keeny  and  Raiffa,  1976;  Sarin,  2000). 

Collaboration  C2  Metrics 

The  following  collaboration  metrics  have  evolved  out  of  work  done  by  Evidence  Based  Research,  Inc. 
for  the  United  States  Office  of  Naval  Research.  These  collaboration  metrics  focus  on  individual  and  team 
cognitive/awareness,  team  behaviour,  and  team  products.  Individual  cognitive  metrics  measure 
collaboration,  team  members’  understandings  about  their  mission  and  their  team,  and  team  cognitive 
metrics  apply  the  individual  cognitive  metrics  to  quantify  the  level  of  awareness  in  a  team.  There  are  four 
classes  of  these  metrics: 

•  Averages  of  the  understanding  among  team  members; 

•  Extent  of  alignment  of  these  understandings; 

•  Maximum  level  of  understanding  anywhere  within  the  team;  and 

•  Presence  of  gaps  in  understanding  throughout  the  entire  team. 

Team  behaviour  metrics  measure  the  key  behaviours  indicative  of  effective  teams.  These  behaviours 
include  smooth  and  efficient  synchronisation,  efficient  information  exchange,  adaptability,  effective 
workload  distribution,  and  team  member  engagement.  Team  product  metrics  measure  product  quality  and 
team  efficiency  and  are  the  bottom-line  “proof  of  the  pudding”  metrics,  applicable  whether  a  team  or  a 
single  individual  produces  the  product. 

Other  metrics  under  consideration  are:  task  performance,  workload,  level  of  engagement  (buy-in), 
synchronisation,  information  needs  workload  and  handling,  workload  awareness  and  handling, 
and  problem  awareness  and  handling. 


5.5  OTHER  CONSIDERATIONS  IN  THE  SELECTION  AND 
INTERPRETATION  OF  MoM 

5,5,1  Effects  of  Uncertainty 

In  order  to  state  a  level  of  confidence  in  the  interpretation  of  MoM,  the  underlying  assumptions  must  be 
clearly  stated  and  uncertainties  recognised.  Uncertainties  manifest  themselves  in  several  ways  that  may 
affect  MoM.  They  may  be  grouped  as  follows: 

Study  assumptions  -  (uncertainties  in  the  scenario,  model  input); 

•  Relevance  to  the  purpose  of  the  evaluation,  uncertainties  in  the  military  objective,  knowledge  of 
enemy  concept  of  operations,  intentions,  capabilities,  weapon  performance,  uncertainties  in  terrain 
data,  etc. 
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Modelling  assumptions  -  (uncertainties  in  the  model,  structural  uncertainty);  and 

•  Human  performance,  parameters,  objects,  attributes,  processes,  effects  of  constraints,  effects  of 
aggregation  and  de-aggregation,  deterministic  (usually  high  hierarchical  level  but  low  resolution) 
versus  stochastic  models,  especially  in  OOTWs;  and 

•  Uncertainties  about  the  implications  of  value  changes  in  lower-level  MoM  with  respect  to  the 
values  of  higher-level  MoM  (e.g.  increases  in  arrest  rates  at  late  stages  of  an  OOTW  may  be 
negative,  while  increases  in  arrest  rates  at  early  stages  may  be  positive). 

Model  sensitivity  -  (uncertainties  in  the  outcome) 

•  Hypersensitivity  to  input  variations,  (instability  or  chaos  theory),  effects  of  model 
non-linearities  and  non-monotonic  behaviour  (effects  of  thresholds),  decisionmaking  for  local 
versus  global  optimisations,  etc.;  and 

•  Sensitivity  analysis  may  be  applied  to  identify  uncertainty.  By  varying  the  assumptions  and  input 
data  within  the  plausible  ranges,  excursions  in  the  analysis  verified  by  the  subject  matter  experts 
provide  insight  into  the  effects  of  uncertainty. 

5,5,2  Impact  of  Technological  Changes 

The  rapid  pace  of  technological  change  involving  information  systems  is  causing  major  changes  in  the 
way  C2  is  perceived  and  executed,  leading  to  potential  changes  in  the  way  war  fighting  commands 
are  organised.  For  example,  the  last  decade  has  seen  the  emergence  of  collaborative  technologies, 
which  enable  new  ways  to  command  and  control  military  forces  and  for  them  to  interface  with  other 
actors.  To  keep  pace  with  and  to  evaluate  the  impacts  of  these  changes,  the  nature  of  these  changes  and 
impacts  need  to  be  understood  so  that  the  appropriate  MoM  can  be  developed. 

One  approach  to  doing  this  postulates  those  differences  likely  to  occur  between  today’s  C2  and  future  C2, 
and  then  describes  an  evaluation  methodology,  including  MoM,  to  measure  the  impact  of  the  changes. 
Other  approaches  are  Social  Construction  of  Technology  (SCOT)  (Bijker,  1989),  Socio-Technical 
Networks  (Elzen,  Enserink  and  Smit,  1996)  and  Actor  Network  Theory  (ANT)  (Eatour,  1997). 


5,6  CONCLUSIONS 

No  single  measure  or  methodology  exists  that  satisfactorily  assesses  the  overall  effectiveness  of  C2 
systems.  As  a  minimum,  the  following  factors  must  be  considered  in  conducting  an  analysis  of  C2: 

•  Determine  the  appropriate  levels  of  MoM  hierarchy; 

•  Identify  specific  MoM  which  are  practically  obtainable; 

•  Specify  means  of  collection  of  MoM; 

•  Assure  the  validity  and  reliability  of  measures  for  correct  interpretation  with  quantifiable  levels  of 
confidence; 

•  Be  aware  that  variation  in  measurements  (e.g.  due  to  human  factors)  may  well  cause  unacceptable 
levels  of  uncertainty.  Hence  the  analyst  must  pay  particular  attention  to  measurements  related  to 
the  human  element; 

•  Consider  that  while  MoPE  and  MoFE  may  provide  the  most  persuasive  measures  from  the 
military  perspective,  MoCE  and  MoP  are  the  most  readily  derivable  by  operations  analysts;  and 

•  Account  for  the  principles  of  reliability  and  validity  to  avoid  the  risk  of  generating  false 
conclusions. 
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5.6.1  Recommendations  for  Generation  and  Selection  of  MoM 

The  principal  objective  for  MoM  is  to  determine  judgements  of  the  degree  to  which  C2  or  changes  to  C2 
may  improve  force  effectiveness  and  to  provide  convincing  arguments  for  the  improvements.  It  is 
important  to  stress  that  the  purpose  is  to  assess  the  contributions  of  C2  in  terms  of  how  C2  improves  the 
effectiveness  of  military  missions,  and  not  the  quality  of  the  C2  process  itself  However,  to  arrive  at  these 
assessments  and  assign  attribution  to  the  C2  system,  the  C2  system  must  be  included  in  the  analysis. 
To  achieve  this  objective,  the  following  steps  are  required: 

•  The  objectives  for  the  assessment  must  be  established  and  clearly  stated; 

•  Selection  of  MoM  should  not  be  done  in  isolation  from  consideration  of  the  assumptions, 
constraints,  models,  tools,  scenarios,  or  other  elements  of  the  analytic  plan  and  assessment 
process.  The  assumptions  used  in  the  model  and/or  evaluation  must  be  stated  along  with  their 
potential  impact  on  the  results; 

•  A  detailed  assessment  of  reliability  and  validity  of  the  selected  measures  needs  to  be  made  in 
order  to  determine  a  level  of  confidence  in  measures;  and 

•  For  C2  acquisition  analyses,  the  generation  of  measures  should  occur  in  parallel  with  the 
development  of  the  system,  so  that  as  the  system  is  being  matured,  developers  can  know  the 
standards  to  which  they  are  being  held. 

5.6.2  Summary  of  the  Challenges  and  Issues  in  the  Evaluation  of  C2 

•  Correlation  of  MoPE  and  MoFE  with  C2  process  measures  (e.g.  battle  outcome  against  lower-level 
measures)  is  difficult. 

•  Separation  and  linkage  of  the  respective  relationships  between  C2  and  users,  organisations  and 
military  objectives  requires  some  effort. 

•  Aggregated  measures  (e.g.  as  obtained  from  ACCES  or  MAUT)  have  limitations  in  the  diagnosis  of 
C2  success  or  failure.  A  careful  analysis  is  required  to  provide  a  comprehensive  assessment  of  highly 
complex  C2  systems  based  on  a  small  number  of  summary  measures  of  outcome  and  process. 

•  The  assessment  of  the  reliability  of  measures  in  an  environment  where  sample  sizes  are  small  will 
remain  difficult  and  may  require  the  use  of  non-parametric  statistics. 

•  The  analyst  must  pay  attention  to  the  complex  task  of  establishing  and  measuring  control  variables  in 
order  to  achieve  correlation  of  measures  against  a  wide  spectrum  of  scenarios  and  staff. 

•  Defining  criteria  to  differentiate  measures  must  be  established. 

•  Verifying  measurement  criteria  (e.g.  discrimination)  must  be  ensured. 

•  For  the  near  future,  collecting  data  to  support  C2  measures  will  remain  labour  intensive  because  C2 
processes  remain  human  intensive. 

•  Many  of  the  measures  for  information  processing  concern  completeness  of  the  information.  Deciding 
what  makes  information  complete  requires  co-ordination  and  co-operation  between  the  assessor  and 
the  user. 

•  The  relationship  between  outcome  and  process  may  be  complex  because  C2  is  an  integrated  system 
with  continuous  feedback. 
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•  The  analysis  of  uncertainties  and  measures  of  central  tendency  and  dispersion  are  both  significant 
when  examining  C2  issues. 

•  A  Command  Post  Exercise  (CPX)  is  a  useful  venue  for  evaluation  of  C2.  However,  the  costs  involved 
generally  preclude  conducting  a  CPX  solely  to  evaluate  the  C2  process  or  a  C2  system.  Cost  control  is 
increasingly  leading  to  the  use  of  laboratory  and  human  in  the  loop  experimentation  to  develop 
knowledge  and  insights. 

5,6.3  Summary  of  the  Challenges  and  Issues  for  OOTW  MoM 

•  Cost/benefit:  cost  in  time  and  effort  for  data  collection  and  analysis  may  outweigh  benefit. 

•  Standardisation  of  data  collection,  evaluation,  and  analysis  with  many  diverse  non-govemmental 
organisations  (NGOs),  UN  agencies,  and  militaries. 

•  Factors  outside  military  control:  other  agencies,  policy  restrictions. 

•  Creation  of  a  rich  and  comprehensive  set  of  MoM  to  preclude  reliance  on  a  limited  number  of  MoM 
as  the  key  to  success. 

•  Availability  and  consistency  of  information  (e.g.,  in  OOTW,  a  possible  consequence  of  different 
factions  controlling  geographic  areas). 

•  Merging  of  strategic,  operational,  and  tactical  domains. 

•  Clear  recognition  of  roles  and  responsibilities  of  all  participants. 


5.7  CHAPTER  5  ACRONYMS 

ACCES 

Army  Command  and  Control  Evaluation  System 

AHWG 

AC/243  Panel  7  Ad  Hoc  Working  Group 

ANT 

Actor  Network  Theory 

C2 

Command  and  Control 

C3I 

Command,  Control,  Communications  &  Intelligence 

COBP 

Code  of  Best  Practise 

CPX 

Command  Post  Exercise 

DP 

Dimensional  Parameters 

HEAT 

Headquarters  Effectiveness  Assessment  System 

MAUT 

Multi-Attribute  Utility  Theory 

MCES 

Modular  Command  and  Control  Evaluation  Structure 

MoCE 

Measures  of  C2  Effectiveness 

MoE 

Measures  of  Effectiveness 

MoFE 

Measures  of  Force  Effectiveness 

MoM 

Measures  of  Merit 

MoP 

Measures  of  Performance 
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MoPE 

Measures  of  Policy  Effectiveness 

MORS 

Military  Operations  Research  Society 

NGO 

Non-Go vemmental  Organisation 

OOTW 

Operations  Other  Than  War 

SCOT 

Social  Construction  of  Technology 

TRAC 

TRADOC  Analysis  Center 

TRADOC 

US  Army  Training  &  Doctrine  Command 
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“If  I  had  time  ...  to  study,  I  think  that  I  should  concentrate  almost  entirely  on  the  ‘actualities  of  war,  ’ 
the  effect  of  tiredness,  hunger,  fear,  lack  of  sleep,  weather  ...  It  is  the  actualities  that  make  war  so 
complicated  and  so  difficult,  and  are  usually  neglected  by  historians.  ”  -Field  Marshall  Archibald  Waved, 
1883  -  1950,  Author  of  Soldiers  and  Soldierins 


6.1  IMPORTANCE  OF  HUMAN  AND  ORGANISATIONAL  FACTORS 

The  human  dimension  largely  distinguishes  command  and  control  (C2).  Key  differences  between  C2 
analyses  and  traditional  military  operations  analysis  (OA)  applications  include  the  need  to  deal  not  only 
with  military  organisations,  but  also  with  distributed  military  teams  (and  organisations)  under  stress  and 
their  decisionmaking  behaviour  as  well.  Moreover,  in  operations  other  than  war  (OOTW),  consideration 
must  be  paid  to  the  behaviour  of  and  interaction  with  non-military  organisations,  political  groupings, 
and  amorphous  groups  such  as  crowds  and  refugees.  Thus,  the  formulation  of  the  problem  and  the 
development  of  solution  strategies  cannot  be  completed  without  explicit  consideration  of  both  human  and 
organisational  issues. 
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The  human  factors  of  interest  fall  into  three  major  categories: 

•  Human  behaviour  related  to  performance  degradation,  such  as  stress  and  fatigue,  and  as  a 
consequence  of  social  interactions  among  individuals  and  members  of  groups; 

•  Decisionmaking  behaviour  (cognitive  questions)  including  the  cognitive  complexity  of  the  issues 
and  the  capacities  of  the  commanders  or  other  decisionmakers  of  interest;  and 

•  Command  style. 

By  contrast,  organisational  factors  deal  with  relationships  among  groups  of  individuals,  including 
connectivity,  roles,  and  organisational  structures. 

Since  both  human  and  organisational  factors  can  affect  C2  performance,  the  operations  analyst  must 
consider  their  impact  early  in  the  research  design  process  and  review  a  priori  assumptions  about  them  in 
an  iterative  manner  throughout  the  entire  analytical  process.  Human  and  organisational  factors  must  be 
considered  as  part  of  structuring  the  problem,  selecting  measures  of  merit  (MoM),  defining  scenarios, 
developing  solution  strategies,  and  selection  of  tools. 

The  first  key  consideration  when  structuring  the  problem  is  whether  individual  decisionmaking  and 
behaviour  of  individuals  or  groups  is  important  to  the  C2  processes  under  analysis.  If  the  research  question 
can  be  answered  without  considering  differences  between  individual  decisionmakers  and  groups  then  the 
additional  complexity  that  issue  introduces  should  be  avoided.  For  example,  in  addressing  a  C2  issue  that 
deals,  all  other  things  being  equal,  with  a  simple  change  in  connectivity  (which  headquarters  will  have 
which  linkages  to  others),  human  behaviour  may  not  be  important  to  the  analysis  at  least  as  long  as  combat 
missions  are  involved.  However,  the  same  change  in  connectivity  might  affect  relations  with  non-military 
organisations  and  individuals  essential  for  mission  success  in  OOTW  suggesting  their  response  must  be 
accounted  for  in  an  appropriate  manner.  Thus,  deciding  on  the  importance  of  human  issues  in  OOTW  and 
identifying  the  issues  at  stake  requires  a  good  general  understanding  of  human  behaviour  and  its 
underlying  motivations  beyond  military  common  sense  as  well  as  knowledge  about  relevant  cultural 
factors  and  the  interests  of  the  parties  involved. 


6.2  HUMAN  FACTORS 

6,2,1  Human  Performance  and  Behaviour 

Human  performance  affects  behaviour  and  vice  versa.  Human  performance  depends  on  psycho- 
physiological  variables  (e.g.  stress,  fatigue,  sleep  deprivation,  hunger,  and  alertness)  and  on  ergonomic 
and  external  factors  limiting  performance  and  behavioural  freedom.  Individual  and  group  behaviour  is  the 
result  of  social  interaction.  It  includes  interactions  by  military  commanders  and  their  troops,  underlying 
psychological  processes  and  factors  (e.g.  fear,  morale,  and  values),  and  the  cultural,  educational, 
and  religious  background  of  individuals.  There  is  significant  historical  evidence  that  inferior  combat 
potential  as  measured  in  terms  of  numbers  of  personnel  and  weapon  systems  may  be  compensated  for  by 
superior  human  performance  in  battle  (Dupuy,  1979). 

Any  time  human  performance  and/or  behaviour  are  at  issue,  parameters  and/or  models  will  be  needed  to 
reflect  those  issues.  For  example,  systems  that  involve  human  activity,  such  as  watch  or  command  centres, 
need  to  be  studied  in  ways  that  reflect  differences  in  C2  performance  that  can  be  traced  to  human 
performance/behaviour  issues.  In  addition,  differences  can  arise  from  experience  or  training,  coalition 
features  (e.g.  language,  national  doctrine,  command  style),  or  service/branch-unique  doctrine  and  practice. 
These  kinds  of  individual  performance  and  behaviour  issues  may  be  modelled  in  two  ways.  They  can  be 
treated  stochastically,  in  a  manner  that  reflects  their  occurrence  in  “real”  systems  depending  on  situational 
factors  (black  box  approach),  or  in  terms  of  process-oriented  behavioural  models,  describing  the 
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psychological  processes  behind  the  observable  behaviour  of  individuals  or  groups  in  a  given  situation 
(Long  Term  Scientific  Study  SAS-017  on  Human  Behaviour  Representation,  Final  Report,  Chapter  2). 
The  decision  that  human  performance  and  behaviour  may  vary  meaningfully  will  have  a  clear  impact  on 
the  choice  of  models  and  analytic  approaches  (e.g.  stochastic  processes  or  action-theoretic  models  such  as 
Norman’s  Activation  Trigger  Scheme  (ATS)  for  simulating  the  dynamics  of  actions  or  reactions) 
(Norman,  n.d.)- 

Where  human  performance  is  considered  to  be  a  meaningful  factor  (e.g.  C2  within  command  centres  when 
wearing  chemical  protection  gear),  some  experimentation  may  be  necessary  to  develop  realistic 
parameters  for  the  impact  on  error  rates  or  the  pace  of  work.  In  other  cases,  such  as  simple  fatigue, 
human  factors  specialists  may  be  able  to  provide  valid  parameters  from  work  in  other  contexts. 
Such  specialists  are  often  valuable  members  of  research  teams.  In  any  case  where  the  workflow  and  work 
rate  within  command  centres  is  relevant,  human  performance  parameters  must  be  considered.  In  addition, 
human  performance  issues  will  have  some  effect  on  decisionmaking.  Error  rates  increase  as  people 
become  tired  and  overloaded,  altering  the  way  they  work  and  the  information  they  consider. 

However,  research  on  how  to  represent  human  behaviour  and  its  impact  on  performance  in  models  is  still 
in  an  early  stage.  This  is  particularly  true  in  the  context  of  OOTW.  Those  operations  require  the 
co-operation  of  non-military  actors.  In  addition,  military  mission  objectives  include  providing  security  and 
assistance  in  the  reconstruction  of  conditions  in  the  theatre  of  operations  and  helping  to  stabilise  the 
situation  to  a  degree  that  ultimately  permits  the  military  forces  to  leave.  The  use  of  force  in  OOTW  is 
limited  in  degree  and  kind  to  responding  appropriately  to  threats  for  defensive  and  protective  purposes  and 
to  coercive  actions  for  enforcing  compliance  with  agreements.  Military  actions  that  would  be  highly 
effective  in  accomplishing  mission  objectives  in  war  might  be  counterproductive  in  OOTW.  In  fact, 
the  value  of  military  actions  in  OOTW  is  not  so  much  a  question  of  physical  effects,  but  rather  how 
military  actions  and  their  physical  effects  are  perceived  by  the  various  actors  and  the  population  in  the 
theatre,  how  the  military  actors  interpret  the  behaviour  of  the  other  actors,  and  the  critical  mission  task 
conditions  (e.g.  political  interest  and  media  attention).  Thus,  a  careful  analysis  of  mission  tasks  by  human 
science  experts  is  indispensable  for  modelling  and  assessment  of  C2  options  in  OOTW  (Baeyer,  2001). 

It  follows  that,  in  addition  to  the  traditional  composition  operations  research/operations  analysis  (OR/OA) 
assessment  teams,  assessors  of  C2  for  OOTW  must  possess  hands-on  experience  with  such  operations  and 
relevant  non-military  organisations  (e.g.  private  aid  organisations).  The  assessment  team  must  also  have 
access  to  experts  from  the  fields  of  political  science,  cultural  anthropology,  demography,  sociology 
(including  media  impact  research),  social  psychology,  and  individual  psychology.  These  experts  will 
contribute  the  expertise  for  diagnosing  the  relevance  of,  and  differences  in,  performance  and  behaviour  of 
actors  and  for  the  formulation  of  hypotheses  for  assessing  the  “human  issue  risk”  of  analysis  results. 
In  addition,  they  may  contribute  to  the  modelling  of  behavioural  processes  in  analysis  tools  used  to  test  C2 
system  sensitivity  via  parametric  variations  of  human  performance  and  behaviour  parameters. 

However,  well-documented  empirical  knowledge  on  human  performance  in  military  operations  is  scarce 
and  little  is  known  about  its  relevance  in  circumstances  other  than  those  prevailing  when  the  underlying 
data  were  collected.  Similarly,  experience  and  systematically  compiled  data  on  behaviour  and  response  of 
individuals  and  groups  to  actions  and  situations  in  OOTW  are  still  limited  and  theories  on  human 
behaviour  are  mostly  untested  in  the  context  of  military  operations.  Therefore,  dealing  with  human  issues 
in  C2  analyses  reduces  to  the  problem  of  addressing  decisions  under  risk  and  uncertainty  when  each  of  the 
C2  design  options  is  tested  for  the  range  of  possible  hypotheses  on  the  implications  of  human  issues  for 
C2  effectiveness. 
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6.2.2  Human  Decisionmaking 

Increasingly,  assessment  teams  have  to  deal  with  issues  where  individual  decisions  are  important.  This  is 
especially  true  for  OOTW  in  which  even  tactical-level  decisions  by  a  lower  level  military  leader  may  have 
strategic  implications  because  of  media  presence.  This  represents  a  major  challenge  because  the  variety  of 
human  behaviours  involved  makes  modelling  decisionmaking  very  difficult.  Fortunately,  there  are  some 
approaches  that  can  be  used  to  cope  with  these  difficulties.  The  correct  choice,  however,  will  depend  on 
the  research  issue(s). 

In  some  cases  the  analyst  is  asked  to  assume  that  decisionmaking  will  follow  established  doctrine  and 
tactics,  techniques,  and  procedures  (TTPs).  In  these  cases,  the  challenge  is  to  craft  a  set  of  rules  or  look  up 
tables  that  reflect  the  existing  guidance  correctly.  Hence,  a  model  that  replicated  the  “correct”  set  of 
decisions  would  be  useful  for  assessing  simple  C2  issues  such  as  the  impact  of  new  communication  and 
information  technology  or  changes  in  connectivity  or  supporting  relationships  within  the  force.  However, 
maximising  the  benefits  of  technical  and  organisational  changes  in  a  C2  system  might  require  an 
appropriate  adaptation  of  doctrine  and  TTPs.  Therefore,  testing  the  impact  of  established  rules  should  be 
part  of  the  analysis. 

Testing  the  impact  of  rules  is  an  indispensable  prerequisite  for  models  that  have  built-in  sets  of  rules  that 
are  not  driven  by  approved  TTPs,  but  rather  by  opinions  of  subject  matter  experts  or  modellers  whose 
rationales  have  been  neither  validated  nor  accredited.  Considerable  knowledge  exists  on  how  to 
organise  and  validate  such  expert  elicitation.  Here  again,  specialised  team  members  may  be  helpful. 
Simple  adoption  of  models  developed  from  subject  matter  experts  will  put  the  assessment  team  at 
considerable  risk  of  accepting  false  conclusions.  When  such  models  must  be  adopted,  they  should  be 
explored  in  detail  to  uncover  their  driving  assumptions  and  subjected  to  sensitivity  analyses  (Chapter  9). 
Where  this  cannot  be  done,  these  models  are  best  avoided  when  C2  assessments  are  performed. 

From  human  factors  research  on  stimuli  that  influence  human  decisionmaking  we  have  learned  that  human 
decisionmaking  capability  is  degraded  in  some  situations  and  enhanced  in  others.  These  stimuli  may 
originate  from,  inter  alia: 

•  BiologicaFphysiological  processes  (e.g.  physical  overexertion  [fatigue],  use  of  bio-chemical 
substances,  and/or  sensory  deprivation); 

•  Cognitive  sources  (e.g.  confusion  arising  in  unfamiliar  or  unknown  situations); 

•  Psychological  processes  (e.g.  processes  causing  emotions  and  stress); 

•  Social  processes  (e.g.  group  dynamics  that  coerce  or  reinforce  individual  decisionmaking 
depending  on  accepted  social  norms,  organisational  infrastructure  and  procedures); 

•  Environmental  factors  (e.g.  darkness,  austere  and/or  uncomfortable  environments);  and 

•  The  decision  support  tools  and  technologies  that  humans  use  (e.g.  information  displays  and 
decision  support  software).  It  is  important  to  look  beyond  technology  at  whether  or  not  human 
decisionmaking  is  improved,  or  even  constrained  in  some  cases,  by  using  computerised  decision 
aids  depending  upon  their  functionality  and  configuration. 


6.3  TYPES  OF  DECISIONS 

The  nature  of  the  decisions  being  supported  by  C2  systems  will  also  enable  the  assessment  team  to  make 
intelligent  decisions  about  how  they  influence  the  analysis.  Three  useful  decision  types  can  be 
distinguished: 

•  Automatable  decisions; 
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•  Contingent  decisions;  and 

•  Complex  decisions. 

6.3.1  Automatable  Decisions 

Automatable  decisions  fall  into  the  category  of  “simple  decisions”.  The  range  of  decision  options  is  finite 
and  known,  and  the  criteria  for  selecting  among  them  are  clear.  Basic  sensor-to-shooter  decisions  are 
simple  decisions  that  are  usually  automated  for  the  sake  of  timely  response  (e.g.  anti-aircraft  or  missile 
defence  systems).  Similarly,  the  selection  of  patrol  routes,  inspection  strategies,  and  many  logistics 
decisions  relevant  for  OOTW  can  be  automated.  For  example,  scheduling  can  be  seen  as  an  optimisation 
problem  in  which  time,  space,  and  priorities  are  traded  off  to  generate  a  “best”  answer.  Even  though  the 
decision  environment  is  constantly  changing  due  to  factors  such  as  weather  and  mechanical  problems, 
scheduling  decisions  are  characterised  by  rules  and  algorithms.  Models  of  automatable  decisions  of  this 
kind  can  be  built  relatively  easily. 

Flowever,  where  the  C2  system  employs  humans  to  make  these  choices,  some  error  rate  parameters  will  be 
needed  if  the  results  are  to  be  meaningful.  For  example,  error  rates  may  increase  if  the  time  available  to 
make  a  decision  is  insufficient  or  physical  demands  induce  fatigue.  Even  where  the  operational  concept 
calls  for  the  use  of  automated  systems,  the  analyst  should  explore  the  quality  of  the  data,  information, 
or  knowledge  used  to  drive  the  process  and  the  likelihood  that  humans  will  be  involved  in  collection  or 
fusion. 

In  these  fully  automatable  decisions  the  assumption  is  that  “to  know  is  to  decide”.  In  these  cases, 
if  uncertainty  were  adequately  reduced,  the  correct  course  of  action  or  decision  would  be  obvious.  In  that 
case,  decision  theory  classifies  the  problem  as  a  “decision  under  certainty.”  These  problems  are  trivial 
except  for  the  determination  of  the  utility  function  in  case  there  are  more  than  two  selection  criteria 
that  need  to  be  considered  and/or  when  constraints  need  to  be  accounted  for  to  complete  the  analysis 
(Keeny  and  Raiffa,  1976). 

6.3.2  Contingent  Decisions 

The  next  level  of  decisionmaking  complexity  is  best  thought  of  as  contingent  decisions.  These  are  cases 
where  the  commander  has  thought  through  the  situation  and  developed  a  set  of  alternative  actions  or 
decisions  that  are  appropriate  to  the  situation,  but  further  information  on  the  operational  environment 
will  be  needed  to  determine  which  is  the  proper  course  of  action.  In  other  words,  “to  know  is  to  decide, 
but  knowing  is  not  yet  possible.”  In  some  NATO  countries  the  research  community  terms  this 
“opportunistic  decisionmaking”. 

In  most  cases  a  lack  of  clear,  precise  knowledge  is  unavoidable.  For  example,  the  commander  in  a 
defensive  posture  may  recognise  that  the  adversary  has  several  potentially  viable  options.  The  adversary 
may  not  even  know  which  alternative  he  will  choose.  In  such  a  case,  the  defending  commander  would 
both  develop  courses  of  action  to  meet  likely  contingencies  and  also  undertake  a  variety  of  information 
collection  activities  designed  to  provide  as  much  warning  as  possible  when  the  attacker  selects  a  main 
attack  option. 

Modelling  contingent  decisions  is  much  more  difficult  than  modelling  automatable  decisions,  but  is 
similar  in  that  an  underlying  set  of  rules  or  algorithms  still  drives  the  process.  The  added  complexity 
comes  from  the  need  to  find  the  time  when  information  is  adequate  to  select  one  of  several  actions. 
The  best  models  for  that  purpose  are  essentially  hypothesis  testing  models.  They  align  information  about 
the  operational  environment  against  a  finite  set  of  alternative  futures  and  perform  probability  calculations 
to  determine  when  the  commander  has  enough  confidence  to  act  or  to  estimate  the  information  gain  in 
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terms  of  the  expected  value  added  to  decisions  as  new  information  arrives  (Sherrill,  1996).  Information 
entropy  is  also  a  valuable  measure  of  the  information  state  of  the  commander.  It  can  also  be  extended  to  a 
measure  of  information  dominance  (Perry  and  Moffat,  1997). 

6,3.3  Complex  Decisions 

Finally,  “complex”  decisions  are  very  difficult  to  model.  These  require  the  decisionmaking  system  to: 

•  Recognise  when  a  decision  needs  to  be  made; 

•  Identify  the  relevant  set  of  options; 

•  Specify  the  criteria  by  which  they  will  be  judged;  and 

•  Determine  when  the  decision  will  be  made. 

Examples  of  complex  decisions  include  the  definition  of  missions  at  the  operational  level,  decisions 
to  change  the  fundamental  activity  of  the  organisation  (e.g.  shift  from  the  offence  to  the  defence), 
and  the  process  that  creates  courses  of  action  in  response  to  events  on  the  battlefield  or  in  OOTW. 
Except  when  doctrinal  answers  are  available,  complex  decisions  are  very  difficult  to  model  and  even  more 
difficult  to  validate  or  accredit.  Most  successful  efforts  dealing  with  complex  decisions  have  used 
“human-in-the-loop”  techniques  and  relied  on  the  quality  and  variety  of  experts  employed  for  reliability 
and  validity.  Some  promising  research  on  modelling  complex  decisions  in  military  operations  has  been 
completed  in  the  UK  and  this  is  in  the  process  of  being  incorporated  as  the  core  of  the  next  generation  of 
closed  form  simulation  models  of  conflict  being  developed  by  Defence  Science  and  Technology 
Eaboratories  (DSTE)  (Moffat,  2000;  Moffat,  2002).  Similar  research  on  tactical  decision  automata  is 
going  on  in  Germany  to  improve  the  capability  of  simulation  systems  for  analysis  as  well  as  training  and 
staff  exercises  support  (Hofmann  and  Hofinan,  2000,  von  Baeyer,  2001). 

Command  Style 

Assessment  teams  often  encounter  the  argument  that  decisionmaking  depends  on  the  “commander’s 
style.”  Moreover,  they  are  told,  systems  must  be  designed  to  support  commanders  with  differing  styles. 
Because  it  is  an  elusive  and  multi-dimensional  concept,  command  style  represents  a  challenge  to 
modelling.  However,  this  factor  can  be  accommodated  if  the  analyst  is  able  to  develop  a  clear  concept  of 
the  alternative  command  styles  that  must  be  recognised  and  their  consequences  for  military 
decisionmaking. 

Attributes  of  the  Commander 

Differences  in  command  style  may  be  reflected  by  appropriate  attributes  such  as  the  background 
and  training  of  commanders,  their  decision  and  order  style,  risk  tolerance,  and  operational  experience. 
For  example,  in  conjunction  with  field  experience,  the  background  and  training  of  commanders  affect  the 
richness  of  their  understanding  of  the  military  situation  and  their  capacity  to  influence  it. 

Organisational  Style 

Another  not  totally  unrelated  topology  deals  with  the  degree  to  which  the  commander  uses  a  formal 
decomposition  of  the  situation  versus  a  holistic,  integrated  vision.  The  decomposition  style  of 
management  is  associated  with  hierarchical  and  segmented  work,  as  in  the  Napoleonic  or  classic  German 
general  staff.  This  heavily  structured  process  allows  centralised  control  and  tight  coupling  between  the 
structure  of  the  problem,  the  structure  of  the  supporting  staff,  and  the  flow  of  information  within  and 
between  command  centres.  The  classic  centralised  commander  imposes  his  style  on  the  C2  process  and 
impacts  key  organisational  issues  as  well  as  decision  style. 
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The  alternative  command  style  is  an  open  and  holistic  one  in  which  senior  staff  and  commanders  from 
related  command  levels  are  directly  involved  in  a  broad  development  of  courses  of  action  and 
implementation  plans.  This  more  open  process  also  has  implications  for  the  information  flow  within  and 
between  command  centres.  While  decisions  are  still  made  authoritatively  at  the  centre  (by  the  commander 
or  senior  staff),  they  tend  to  generate  loose  guidance  (mission  type  orders)  and  to  enable  lower  level 
commanders  and  their  staffs  more  latitude  in  implementation. 

Risk  Style 

Another  topology  applied  by  some  practitioners  is  the  degree  to  which  individual  commanders 
(and  doctrines)  are  risk  averse  versus  attracted  to  risk.  Most  military  enterprises  have  some  properties  that 
impel  commanders  to  minimise  risk.  The  fact  that  lives,  national  treasure,  and  serious  national  interests  are 
involved  in  warfare  suggests  that  risk  averse  strategies  will  tend  to  dominate.  However,  some  military 
commanders  are  more  comfortable  with  greater  risk.  Indeed,  outnumbered  or  otherwise  disadvantaged 
forces  must  often  take  risks  in  order  to  prevail.  To  the  extent  that  the  relative  risk  aversion  of  commanders 
is  relevant  to  the  C2  analyses  underway,  assessment  teams  will  need  to  define  and  model  variables  that 
represent  this  factor  (Schultz,  n.d.). 

Recent  research  on  Bayesian  decisionmaking  had  indicated  that  the  perception  of  risk  as  measured  by 
utility  loss  relative  to  a  goal  value  and  perception  of  future  outcomes  can  in  combination  give  insight  into 
the  way  these  affect  command  decisionmaking  (Moffat,  2002). 

Orders  Style 

Commanders,  and  national  command  styles,  have  also  been  shown  to  differ  in  the  degree  of  detail 
contained  in  directives  to  subordinates.  At  one  end  of  the  spectrum  is  the  commander  who  issues  detailed 
orders  that  specify  what  is  to  be  done,  how  it  is  to  be  accomplished,  and  when  and  where  the  specified 
activities  are  to  occur.  At  the  opposite  end  of  the  spectrum  is  the  commander  who  issues  “mission  type 
orders”  which  simply  specify  the  mission  to  be  accomplished  and  leave  decisions  about  the  detailed 
objectives,  forces  to  be  employed,  critical  terrain,  and  timing  up  to  the  subordinate  commanders. 
In  between  are  those  who  specify  a  series  of  linked  objectives  (cross  the  river,  take  the  high  ground  in  the 
north,  and  be  prepared  to  defend  or  carry  the  attack  north-east  into  the  valley)  and  supporting  detail 
(e.g.  forces  available,  rough  timetable  keyed  to  the  objectives)  but  leave  subordinates  with  considerable 
discretion  within  that  guidance.  Both  the  speed  of  the  C2  process  and  the  distribution  of  C2  work  across 
command  centres  (particularly  planning  and  operations  management)  will  vary  greatly  depending  on  the 
commander’s  style  on  this  dimension.  National  doctrine  and  practice  may  also  influence  this  factor. 

Other  Typologies 

Other  typologies  of  command  styles  are,  of  course,  possible  and  may  be  more  relevant  to  particular  C2 
analyses.  Human  behaviour  experts  (e.g.  cognitive  and  organisational  psychologists  and  anthropologists) 
should  be  recruited  to  the  project  team  if  novel  categories  are  developed.  However,  the  most  important 
issue  when  dealing  with  command  style  is  whether  it  is  included  in  the  analysis  at  all.  The  C2  research 
related  hypotheses  under  analysis  should  dictate  the  forms  of  command  style  examined.  However, 
the  impact  of  command  style  should  only  be  examined  because  it  appears  to  be  necessary  to  answer  the 
analytic  question(s)  of  interest.  Otherwise  it  tends  to  introduce  a  level  of  complexity  that  may  confound 
the  other  analyses  underway. 


6.4  ORGANISATIONAL  FACTORS 

All  elements  of  the  C2  system  are  ultimately  related  to  one  another.  The  linkage  between  human  and 
organisational  issues,  however,  is  particularly  direct  and  close.  Properly  done,  organisational  design 
reflects  the  interaction  among  the  tasks  to  be  done,  the  people  available  to  perform  them,  and  the  systems 
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or  tools  that  support  those  people.  Henee,  the  “proper”  organisation  of  C2  depends  in  large  measure  on  the 
eapabilities,  training,  and  experienee  of  the  people  in  the  C2  system. 

Organisation  is  a  serious  subject  in  military  analyses.  For  centuries  the  military  has  sought  to  implement 
unambiguous  relationships  and  responsibilities.  Unity  of  command  is  a  central  principle  of  war.  When  it 
has  been  lost  or  comes  into  question,  as  in  OOTW,  the  professional  militaries  of  the  world  have  found 
themselves  very  uncomfortable.  Fortunately,  military  organisational  issues  are  driven  by  a  fairly  small  and 
finite  list  of  principles.  Assessment  teams  asked  to  work  on  C2  issues  can  use  the  known  list  of  factors  as 
a  checklist  about  organisational  differences  to  determine  whether  they  need  to  build  organisational  matters 
into  their  research  designs.  This  includes  the  issue  of  informal  relationships  that  may  have  evolved 
in  order  to  overcome  organisational  deficits  and  thus  streamline  day-to-day  operations.  In  fact, 
an  organisational  design  and  command  style  that  are  supportive  of  building  informal  relationships  may 
provide  the  flexibility  for  efficiently  handling  the  manifold  demands  facing  commanders  in  OOTW. 
Also,  it  should  be  recognised  that  organisational  implications  which  are  perceived  as  detrimental  to  the 
interests  of  the  affected  individuals  and  groups  inevitably  would  jeopardise  co-operation,  technical  and 
procedural  improvements  in  C2  notwithstanding. 

6,4,1  Organisational  Differences 

The  principal  differences  between  military  organisations  are  related  to  structure,  function,  and  capacity. 
Any  change  or  innovation  that  can  be  introduced  in  a  C2  organisation  falls  into  one  of  these  three 
categories  which,  therefore,  may  be  used  to  guide  analysts  when  structuring  a  problem. 

Structural  differences  include: 

•  The  number  of  echelons  or  layers  in  the  command  structure; 

•  The  span  of  control  for  nodes  in  the  command  structure; 

•  The  pattern  of  linkages  between  those  nodes  (e.g.  hierarchical,  spokes  of  a  wheel,  multi- 
connected,  networked); 

•  Permanent  versus  transitory  organisational  relationship;  and 

•  Formal  versus  informal  relationships. 

Functional  differences  include: 

•  The  distribution  of  responsibility:  where  functional  activities  are  located  (e.g.  intelligence, 
logistics,  command,  civil  military  co-operation  [CIMIC]); 

•  The  distribution  of  authority  (ideally  co-located  with  responsibility); 

•  The  distribution  of  information; 

•  Functional  specificity  (e.g.  fire  support  vs.  infantry  or  close  air  support  vs.  defensive  counter  air) 
vs.  general  and  integrated  military  capabilities  (mission  tailored  task  forces);  and 

•  Degree  of  ambiguity  in  command  relationships. 

Difference  in  capacity  are  related  to  differences  in: 

•  Personnel  (e.g.  quality,  training,  experience); 

•  Communications  systems  and  architectures; 

•  Information  processing  systems  and  architectures;  and 

•  Field  training  and  operational  experience. 
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All  these  dimensions  ean  be  modelled,  some  more  easily  than  others.  However,  the  assessment  team’s 
ehallenge  is  to  identify  those  organisational  faetors  that  are  relevant  to  the  C2  analyses  underway. 
This  issue  must  also  be  addressed  knowing  that  organisational  faetors  are  interrelated:  changing  one  may 
change  others.  For  example,  the  decision  to  eliminate  a  level  of  hierarchy  within  a  military  organisation 
may  have  a  profound  influence  on  the  span  of  control.  Similarly,  changing  the  distribution  of  information 
so  that  it  no  longer  follows  the  chain  of  command  may  have  profound  implications  for  the  ambiguity  of 
command  relationships.  Similar  effects  can  be  expected  when  coalition  operations  involve  ad  hoc 
members. 

6,4,2  Treatment  of  Organisational  Factors 

Because  of  the  large  numbers  of  organisational  variables  that  may  be  relevant  to  the  analysis  of  C2  issues, 
they  must  be  approached  carefully  and  systematically.  When  possible,  organisation  theory  expertise 
should  be  brought  into  the  assessment  team.  Review  of  organisational  issues  is  treated  in  a  two-step 
process  guided  by  a  hypothesis  testing  logic.  The  first  step  should  assess  whether  any  organisational 
variable  is  being  manipulated  directly.  For  example,  a  decision  to  move  from  warfare  domain  task  forces' 
to  mission  tailored  task  forces  with  air  and  land  units  planning  and  operating  together  under  a  joint 
commander  would  be  seen  as  a  direct  manipulation  of  organisational  factors  and  should  be  studied  as 
such.  The  second  step,  a  search  for  indirect  effects  of  organisational  factors,  may  be  more  difficult  and 
will  require  that  the  assessment  team  use  the  list  of  possible  factors  as  a  checklist  and  think  through 
whether  they  may  be  altered  in  a  prepositional  (if,  then)  logic.  An  assessment  team  that  posits  a 
relationship  between  the  C2  analysis  and  an  organisational  issue  should  be  able  to  make  a  clear  statement 
of  the  hypothesis  and  the  causality  anticipated.  This  will  enable  the  research  design  to  cover  not  only  the 
gross  effect  anticipated,  but  also  the  underlying  causal  mechanism(s)  that  will  be  present  if  the  proposition 
is  correct.  Adopting  this  hypothesis  approach  is  also  a  safeguard  against  assuming  that  organisational 
issues  are  easily  or  well  understood  and  can  be  treated  by  assumption.  In  fact,  the  organisational  arena, 
like  that  of  human  factors,  is  one  of  the  most  difficult  in  C2  analysis  and  must  be  approached  with  care 
and  rigour. 

For  example,  the  small  group  literature  makes  a  clear  prediction  that  multi-connected  groups  will  be  able 
to  generate  better  answers  to  complex  problems,  but  will  take  more  time  to  do  so  than  either  hierarchies  or 
star  shaped  groups.  The  causal  mechanism  in  that  theory  is  greater  dialogue  and  the  representation  of  more 
independent  viewpoints.  Moreover,  these  richer  discussions  are  expected  to  take  more  time.  All  other 
things  being  equal,  multi-connected  groups  that  are  found  to  generate  better  answers  to  complex  problems 
should  also  engage  in  more  dialogue  and  be  found  to  have  considered  more  information  and  or  solutions. 
Modellers  who  want  to  take  advantage  of  this  factor  to  explore  alternatives  to  traditional  hierarchical 
military  decisionmaking  must  also  include  the  negative  features  in  their  C2  models  (e.g.  demands  for  more 
time  from  already  overburdened  staffs  and  slower  decisionmaking). 

Roles 

The  concept  of  a  role  comes  from  sociology.  A  role  is  a  set  of  behaviours  expected  by  the  self  and  others. 
For  most  military  systems  the  roles  of  commander  and  key  staff  are  well  understood  and  arise  from  a 
combination  of  tradition,  training,  experience,  and  rational  planning.  Because  of  their  origin,  roles  are 
often  a  convenient  way  of  capturing  the  doctrine  about  responsibilities  within  the  C2  system. 

Roles  can  be  used  to  capture  “syndromes”  or  sets  of  related  attributes  within  a  C2  system.  For  example, 
an  object  oriented  program  might  have  different  functional  organisations  and  their  leaders  might  be 
defined  as  having  different  attributes  that  reflect  their  decisionmaking  responsibilities  and  the  information 
they  would  receive  or  be  able  to  obtain  from  the  information  network.  When  assessing  new  C2  systems, 
analysts  will  often  need  to  search  for  potential  role  gaps  or  role  overlaps.  Either  of  these  would  be 


'  Land  warfare  with  one  commander,  air  warfare  with  another,  each  reporting  directly  to  a  joint  commander. 
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dysfunctional  in  military  operations  over  time.  Changes  in  information  structures  also  have  considerable 
potential  for  creating  problems  of  this  type. 

Role  gaps,  role  overlaps,  and  even  role  conflicts  may  be  more  of  a  problem  in  OOTW  because  military 
organisations  have  to  assume  new  roles  not  embedded  in  their  tradition  and  experience  and  requiring  them 
to  cooperate  with  a  variety  of  non-military  actors  and  organisations  pursuing  their  particular  objectives 
and  the  roles  of  which  are  ill  defined  in  many  cases.  Thus,  rather  than  being  a  source  of  friction,  changing 
or  adapting  existing  information  structures  may  be  part  of  the  decision  problems  that  the  C2  system  must 
address  in  order  to  eventually  bring  about  a  secure  and  stable  situation  in  the  theatre  of  operations  through 
overarching  “unity  of  command”. 


6.5  HUMAN  AND  ORGANISATIONAL  ISSUES  AND  TECHNOLOGY 

The  foregoing-discussion  of  human  and  organisational  issues  revealed  that  human  performance  and 
behaviour  as  well  as  the  organisational  design  of  C2  systems,  and  therefore  the  effectiveness  of  C2, 
depend  on  the  available  communication  and  information  technology.  In  fact,  a  C2  organisation  resembles 
a  system  composed  of  interacting  human,  organisational,  and  technological  elements  as  depicted 
schematically  in  Figure  6.1  adapted  from  Mandeles  et  al  (1996). 

Figure  6.1  is  meant  to  illustrate  that  the  character  and  performance  of  a  C2  system  may  change  as  anyone 
of  elements  in  these  three  categories  changes.  Moreover,  since  the  human,  organisational  and 
technological  elements  are  closely  linked  in  most  cases,  optimising  each  one  of  them  at  a  time  under 
ceteris  paribus  assumptions  for  the  other  two  rarely  ever  results  in  an  efficient  C2  system  (Schot  and  Rip, 
1996). 


Figure  6.1:  Representation  of  the  C2  System,  based  on  Mandeies  et  ai  (1996). 


In  particular,  the  assessment  of  the  human-technology  relationship  is  a  critical  requirement  that  implies 
challenges  that  can  be  both  social  and  technical  in  nature.  Without  adapting  human  thought  and  behaviour 
patterns  and  organisational  structures  it  may  be  impossible  to  exploit  the  potential  of  new  technology. 
On  the  other  hand,  the  performance  of  new  communication  and  information  technologies  may  exceed 
human  capabilities  of  processing  information  (information  overflow)  and  thus  result  in  a  degradation  of 
human  performance  and  overall  effectiveness  of  a  C2  system,  the  improvement  of  technical  parameters 
notwithstanding. 

The  challenges  of  adapting  technological  capabilities  to  meet  human  capabilities  and  the  requirements  of 
the  social  interaction  processes  of  commanders  and  staff,  and  non-military  actors  and  populations  in 
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OOTW,  require  soeio-teehnieal  assessment  approaehes  of  the  kind  that  evolved  in  the  fields  of  scienee  and 
teehnology  studies  (STS),  technology  assessment  (TA)  (Rip.  1995),  and  constructive  technology 
assessment  (CTA)  (Van  de  Poel,  1999). 


6.6  INTEGRATED  ANALYSES 

Because  the  issues  arising  from  human  and  organisational  factors  are  so  complex  and  so  tightly  coupled, 
C2  assessment  teams  often  use  integrating  tools  to  define  the  key  dimensions  relevant  to  their  analyses  and 
explore  the  relationships  between  and  among  them.  Integrating  tools  are  those  that  use  selected  key  factors 
with  powerful  influence  to  cut  through  the  clutter  and  detail  implied  by  trying  to  study  everything  and 
concentrate  instead  on  the  most  important  elements  in  the  problem.  These  key  driving  factors  are  used  to 
conduct  a  simpler  analysis  that  can  then  be  augmented  by  sensitivity  analyses  and  analytic  excursions  to 
ensure  that  the  problem  has  been  fully  and  properly  understood. 

For  example.  Figure  6.2  has  been  used  to  illustrate  the  relationship  between  the  time  available  to  make  a 
decision,  the  complexity  of  the  decision,  and  the  uncertainty  of  the  information  available  about  the 
situation.  These  three  factors  also  reflect  the  risk  or  opportunity  inherent  in  a  military  situation.  The  more 
complex  a  situation,  the  less  time  available,  and  the  greater  the  uncertainty  of  the  available  information, 
the  greater  the  risks  (and  opportunities)  present. 


Figure  6.2:  Decisionmaking  Drivers. 

While  each  of  these  three  dimensions  can  be  examined  independently,  considerable  insight  can  be  derived 
from  examining  them  as  a  related  set.  This  examination  normally  begins  as  an  exercise  in  hypothesis 
generation,  but  can,  as  research  is  accomplished,  be  converted  into  a  component  of  a  knowledge  base. 
That  is,  as  evidence  confirming  or  calling  for  revising  the  key  hypotheses  is  generated,  the  graphic 
becomes  a  way  of  conveying  known  relationships  and  generating  new  propositions  about  regions  or 
subspaces  that  have  not  yet  been  examined  empirically. 

In  some  sense,  one  comer  of  this  cube  represents  the  worst  of  all  C2  worlds  -  almost  no  time  available,  an 
enormously  complex  problem,  and  considerable  uncertainty  about  the  situation.  Past  research  suggests  that 
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when  these  conditions  exist  the  decisionmaker  has  no  choice  except  to  use  “best  professional  judgement” 
to  match  the  operational  situation  to  some  class  of  well-understood  military  situations  and  act  accordingly 
(Hayes,  1994).  However,  decisionmaking  theory  also  indicates  that  the  wise  commander  will  take 
short-term  actions  designed  to  create  more  time  and/or  more  information  and  thereby  relocate  the  problem 
to  a  “better”  portion  of  the  space.  A  “risk  averse”  commander  will  clearly  attempt  this  transformation  of 
the  situation.  However,  a  more  risk  oriented  leader  may  attempt  to  cut  through  the  fog  of  war  with 
decisive  action. 

The  opposite  comer  of  this  analytic  space,  defined  as  ample  decision  time  available,  limited  complexity, 
and  low  uncertainty,  provides  the  ideal  situation  for  decomposition  of  the  problem  and  development  of 
“optimal”  military  plans.  Many  innovations  in  C2  systems  are  designed  to  move  the  situations  facing 
commanders  of  friendly  forces  toward  this  region.  Indeed,  Van  Creveld’s  analysis  of  C2  defines  it  as  a 
search  for  greater  certainty  (Van  Creveld,  1985). 

This  cube  also  emphasises  an  imperfectly  understood  dimension  of  C2  systems  and  the  decisionmaking 
they  imply.  That  dimension  is  the  speed  at  which  the  situation  is  changing  (the  pace  of  operations) 
in  relation  to  the  time  required  to  make  and  implement  a  military  decision  (the  speed  of  the  C2  system). 
Where  the  speed  of  the  C2  system  is  faster,  proactive  decisions  are  possible.  When  the  pace  of  operations 
is  faster,  decisions  must  be  reactive.  The  commander  who  is  capable  of  making  decisions  that  transform 
the  operation  from  reactive  to  proactive  is  rare  and  enjoys  vision  not  only  about  what  is,  but  also  about 
what  is  possible. 

This  key  relationship  (pace  of  operations  to  speed  of  the  C2  system)  is  the  driving  force  behind  the 
observe,  orient,  decide,  act  (OODA)  loop  and  the  resulting  guidance  to  seek  to  “turn  inside  the  enemy’s 
C2  loop.”  However,  C2  analysts  must  constantly  discipline  their  analyses  away  from  assuming  that 
speed  alone  is  a  desirable  attribute  of  a  C2  system  or  organisation.  Making  and  implementing  bad 
decisions  quickly  will  result  in  more  rapid  failures,  not  military  success.  As  is  discussed  in  the  Chapter  5  - 
Measures  of  Merit  in  detail,  multiple  dimensions  of  performance  need  to  be  analysed  whenever  C2 
systems  are  assessed.  However,  this  requirement  to  look  at  multiple  dimensions  in  order  to  assess  C2  does 
not  obviate  the  value  of  performing  integrated  analyses  of  human  factors  and  organisational  issues. 

6.7  CONCLUSIONS 

•  Issues  of  human  performance  and  behaviour  should  be  incorporated  in  models  used  to  analyse  issues 
that  require  human  activity  either  in  the  form  of  performance  parameters  or  appropriate  sub-models  on 
behaviour. 

•  Decisionmaking  that  is  rule  or  algorithmically  based  can  be  modelled  directly,  but  error  rates  should 
be  estimated  if  humans  are  involved  in  the  relevant  decisionmaking. 

•  Simple  decisions  are  programmable  (with  appropriate  error  rates),  but  also  require  estimations  of 
when  decision  would  be  made. 

•  Complex  decisions  can  be  treated  with  “human  in  the  loop”  tools  and  techniques,  but  new  techniques 
are  being  developed  and  applied  (see  Chapter  5). 

•  Style  of  command  and  decisionmaking  should  be  considered  in  C2  analyses  that  focus  on  specific 
decisionmaking. 

•  Organisational  issues  can  be  decomposed  into  constituent  elements  for  analysis. 

•  Hypotheses  or  propositional  structures  are  often  the  most  useful  approach  to  human  factors  and 
organisational  issues. 
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•  Integrated  analyses  involving  roles  or  seleeted  aspeets  of  a  problem  space  often  provide  a  cohesive 
approach  to  the  complexity  inherent  in  human  factors  and  organisational  issues. 

•  Research  in  organisations  and  human  factors  is  expanding  and  analysts  are  advised  to  consult  the 
available  literature.  Experts  in  this  area  should  be  included  on  the  interdisciplinary  C2  assessment 
teams. 

•  Operational  knowledge  of  human  issues  is  still  weak  in  many  areas.  Systematic  effort  is  required  for 
organising  a  consistent  program  for  experiments  on  human  issues. 


6.8  RECOMMENDATIONS 

•  Human  and  organisational  issues  are  not  closed  topics  and  should  be  considered  early  in  the  process  of 
C2  analysis  when  the  problem  is  formulated  and  a  strategy  is  adopted. 

•  Test  the  impact  of  established  decisionmaking  rules  that  reflect  existing  guidance  as  part  of  an 
analysis.  This  is  an  indispensable  prerequisite  for  models. 

•  The  assessment  team  should  explore  the  quality  of  data,  information,  or  knowledge  used  to  drive  the 
automatable  decisionmaking  process. 

•  Early  on,  the  assessment  team  should  establish  working  relationships  with  the  potential  subjects  of  the 
study  but  be  careful  not  to  allow  this  to  introduce  a  bias. 

•  Human  factors  and  organisational  expertise  should  be  included  in  all  C2  assessment  teams; 
at  least  until  a  decision  can  be  made  that  they  are  not  major  elements  of  the  analysis. 

•  Separate  human  performance  issues  (e.g.  stress  and  fatigue)  from  cognitive  issues 
(e.g.  decisionmaking)  when  possible,  but  recognise  that  they  interact. 

•  Use  a  checklist  (Annex  D)  and  hypothesis-testing  logic  for  reviewing  human  and  organisational 
issues.  Remember  that  human  and  organisational  issues  may  interact  as  do  the  structural,  functional, 
and  capacity  arenas  of  organisations. 

•  Integrated  analytic  tools  that  focus  on  key  variables  that  drive  human  factors  and  organisational  issues 
will  often  prove  useful  in  simplifying  analysis. 

•  Sensitivity  analyses  are  particularly  important  when  working  with  human  factors  and  organisational 
issues. 

•  Experiments  for  testing  hypotheses  on  human  behaviour  underlying  the  C2  analysis  are  strongly 
recommended. 

6.9  CHAPTER  6  ACRONYMS 

ATS  Norman’s  Activation  Trigger  Scheme 

C2  Command  and  Control 

CIMIC  Civil-Military  Co-operation 

CISS  Center  for  Information  Systems  Security 

CTA  Constructive  Technology  Assessment 
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DISA 

Defence  Information  Systems  Agency 

DSTF 

Defence  Science  and  Technology  Faboratories 

MoM 

Measures  of  Merit 

OA 

Operations  Assessment 

OODA  loop 

Observe,  Orient,  Decide,  Act 

OOTW 

Operations  Other  Than  War 

OR/OA 

Operations  Research/Operations  Assessment 

STS 

Science  and  Technology  Studies 

TA 

Technology  Assessment 

TTP 

Tactics,  Techniques,  and  Procedures 

VTC 

Video  Teleconference 
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“Before  beginning  operations  you  must,  without  any  indulgence  or  self-deception,  examine  objectively 
every  step  that  the  enemy  might  undertake  to  thwart  your  plan,  and  consider  in  each  conceivable  case 
what  means  are  open  to  you  to  fulfil  your  goal.  The  more  you  anticipate  the  difficulties  in  advance, 
the  less  surprised  you  will  be  should  you  encounter  them  during  the  campaign.  Besides,  you  have  already 
thought  about  these  obstacles  deliberately,  and  with  composure  you  have  perceived  the  means  of  avoiding 
them,  so  nothing  can  surprise  you.  ’’  -Frederick  the  Great 

“Plans  that  assume  the  likelihood  of  one  particular  world  run  the  risk  of  being  seriously  wrong.  ” 
—James  A.  Dewar,  Carl  H.  builder,  William  M.  Hix,  Morlie  FI.  Levin  from  Assumption-Based  Planning  — 
A  Planning  Tool  for  Very  Uncertain  Times 


7.1  PURPOSE  OF  SCENARIOS 

The  report  of  the  NATO  Panel  7  Ad  Hoe  Working  Group  on  the  Impact  of  C3I  on  the  Battlefield  gave 
extensive  consideration  to  the  role  of  scenarios  in  operations  analysis  (OA).  This  chapter  builds  on  that 
material  to  discuss  the  role  of  scenarios  in  Command  and  Control  (C2)  analysis.  Figure  2.2  in  Chapter  2 
shows  the  role  played  by  scenarios  in  the  overall  C2  assessment  process.  The  analysts  craft  a  set  of 
scenarios  to  provide  the  context  or  environment  for  the  conduct  of  the  operational  analysis.  The  scenarios 
bound  the  arena  of  the  analysis  and  are  used  by  the  analyst  to  focus  the  analysis  on  central  issues. 
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7.2  DEFINITIONS 

These  definitions  are  of  particular  relevance  to  this  Code  of  Best  Practice  (COBP). 

7.2.1  Scenario 

A  description  of  the  area,  the  environment,  means,  objectives,  and  events  related  to  a  conflict  or  a  crisis 
during  a  specified  time  frame  suited  for  satisfactory  study  objectives  and  the  problem  analysis  directives. 

As  described  in  Chapter  4,  scenarios  consists  of  four  elements  -  a  context  (e.g.  a  characterisation  of  a 
geopolitical  situation),  the  participants  (e.g.  intentions,  capabilities  of  blue,  red,  others),  the  environment 
(e.g.  natural  -  weather  and  manmade  -  mines),  and  the  evolution  of  events  in  time.  In  C2  assessments, 
the  purpose  of  scenarios  is  to  ensure  that  the  analysis  is  informed  by  the  appropriate  range  of  opportunities 
to  observe  the  relevant  variables  and  their  interrelationships. 

7.2.2  Approved  Scenario 

In  order  to  support  the  analysis,  the  use  of  an  approved  scenario  is  more  relevant  than  to  analyse  a  current 
situation  or  future  plans.  The  impact  of  all  the  elements  is  more  simple  to  analyse. 

In  some  countries  these  scenarios  are  mainly  developed  by  a  Strategic  Committee  and  are  designed  to 
meet  the  government  strategy.  The  first  step  is  to  implement  a  steering  group,  composed  of  representatives 
of  the  government  and  members  of  the  Armed  Forces,  which  has  the  responsibility  to  describe  all  the 
commitments  or  the  precise  regions  in  which  any  major  event  may  have  an  impact  on  the  foreign  politic  or 
on  the  economy.  This  basic  work  is  then  forwarded  to  a  larger  group  or  a  next  military  level  which  shares 
the  responsibility  for  implementing  the  main  recommendations  in  a  more  strategic  field.  These  scenarios 
specify  the  elements  that  must  be  kept  in  mind  in  the  generic  planning  process  or  in  the  operational  studies 
in  relation  with  the  use  of  military  forces.  These  scenarios  also  describe  the  global  threat,  the  geographic 
areas,  the  political  and  military  objectives  and  the  level  of  force  commitment. 

With  this  basic  approach  done,  the  analyst  or  his  military  counterpart  has  only  to  adapt  those  guidelines, 
to  compare  his  study  to  the  scenarios  and  to  propose  the  more  relevant  alterations  to  his  sponsor  and  his 
assessment  team  (See  2.B.I.). 

At  this  stage  the  decision  to  keep  this  scenario,  to  diverge  or  to  choose  another  one  will  be  discussed 
amonga  1  the  stakeholders.  The  products  of  this  process  are  called  approved  scenarios. 

After  that,  at  any  time  the  analysts  may  develop  vignettes  using  the  approved  scenarios  as  a  base  in  order 
to  focus  on  a  specific  issue.  These  vignettes  can  be  used  as  small  scenarios  to  explore  a  particular  topic. 

The  use  of  prepared  scenarios  must  not  be  considered  as  a  limitation  for  customers  or  analysts  because  all 
the  data  could  be  adapted  by  the  customer  or  the  assessment  team,  the  only  limitation  is  to  avoid  the  re-use 
of  this  particular  scenario  for  a  study  not  in  close  relation  with  the  original  one. 

7.2.3  Planning  Scenario 

A  planning  scenario  is  one  in  which  these  elements  are  defined: 

•  Time  frame  of  the  analysis; 

•  Geographic  area; 

•  Meteorological  environment; 

•  Political,  historical,  economic,  and  social  context; 
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•  Mission  objectives  and  constraints; 

•  Level  of  threat; 

•  Friendly  forces  (and  links  between  each  members  of  the  coalition); 

•  Adversary  forces  (e.g.  enemy  order  of  battle); 

•  Neutral  or  uncommitted  forces; 

•  Non-combatant  (non-govemment  organisations  [NGOs],  international  organisations,  etc.)  and 
other  relevant  actors;  and 

•  Media  (when  relevant). 

7,2,4  Operational  Scenario 

An  operational  scenario  contains  additional  details  especially  with  respect  to  threats,  orders  of  battle, 
tactics,  rules  of  engagement,  courses  of  action,  deployment,  end  state  and  reserves. 


7,2,5  Vignette 

The  term  “vignette”  is  sometimes  used  for  a  scenario  that  is  not  approved.  The  term  is  also  primarily  used 
for  smaller  scenarios,  particularly  as  excursions  from  the  main  scenario. 


7,3  ROLE  OF  SCENARIOS  IN  C2  ANALYSIS 

In  general,  the  ideal  OA  is  scenario  independent.  All  relevant  factors  can  be  identified  and  dealt  with 
empirically  and  algorithmically  across  a  range  of  military  contexts.  However,  C2  involves  human 
behaviour,  organisations,  missions,  and  other  complex  phenomena.  Human  behaviour  is  very  difficult  to 
put  into  equations  (see  “Human  Decisionmaking”  in  Chapter  6).  There  is  no  single  linear  dimension  for 
organisations  or  human  issues.  Moreover,  military  missions  do  not  form  simple  dimensions.  Therefore, 
for  most  C2  analyses,  the  context  must  be  defined.  This  is  the  role  for  which  the  analyst  defines  the 
scenarios. 

The  formulation  of  the  original  problem  dictates  the  contents  of  the  scenarios.  There  are  no  overall 
scenarios  that  are  independent  of  a  specific  problem.  Scenarios  are  never  truly  generic,  but  rather  are 
customised  if  only  by  the  assumptions  built  into  them.  The  boundaries  of  the  scenario  space  should  be 
defined  in  part  by  the  issues  unique  to  the  problem  under  analysis. 

Organisational  issues  include  the  involvement  of  various  levels  of  military  and  non-military  hierarchies, 
including  different  command  levels.  This  requires  that  scenarios  accommodate  analysis  across  different 
echelons  of  command.  Information  processing  and  the  characteristics  of  information  must  also  be 
accommodated.  Human  factors  include  the  decisionmaking  process  and  supporting  staff  activities. 

The  analyst  will  need  to  design  or  select  scenarios  to  address  C2  under  a  broad  range  of  circumstances. 
This  taxonomy  of  C2  analysis  might  include: 

•  Defence-planning; 

•  Force  structure  and  organisation; 

•  Mission  analysis; 

•  Doctrine/tactics  development; 

•  Cost-benefit/effectiveness  analysis; 
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•  Training  and  education; 

•  Balancing  C2  systems  and  weapon/sensor  systems; 

•  C2  system  procurement,  which  will  often  require  more  detailed,  task  specific  scenarios  to  cover 
the  range  of  relevant  system  uses;  and 

•  Non-combatant  actor’s  access  to  C2  systems  and  integration  into  the  information  flow,  typically 
through  liaison  officers,  but  increasingly  through  a  variety  of  other  means. 

In  essence,  the  role  of  a  scenario  is  to  define  a  set  of  conditions  and  restrictions  to  enable  “good”  analysis 
as  well  as  to  create  a  structure  within  which  the  results  of  the  analysis  can  be  understood  and  interpreted. 

7.4  UNDERSTANDING  AND  INTERPRETING  THE  RESULTS  OF 
OPERATIONAL  ANALYSIS 

The  analyst  uses  scenarios  to  understand  and  interpret  the  value  of  OA  study  results  for  the  focus  of  the 
analysis.  The  scenarios  provide  the  context  in  which  the  C2  system  will  be  assessed.  It  should  reflect  the 
scenarios  envisaged  by  the  originator  of  the  requirement  for  the  system.  Often  artificial  constraints  must 
be  introduced  into  the  scenarios,  due  to  cost  considerations,  to  properly  focus  the  analysis,  or  both. 
The  scenario  developer  must  have  an  appreciation  of  the  objectives  of  the  simulation,  experiment, 
or  exercise  analysis  plan  in  order  to  determine  the  artificial  constraints  necessary  to  facilitate  the  analysis. 
The  analyst  needs  to  be  aware  of  scenario  assumptions  and  artificial  constraints. 

There  are  several  essential  questions  in  C2  analysis  that  should  be  addressed  in  the  scenario 
considerations: 

•  Operational  benefits  of  C2  to  be  translated  in  the  definitions  of  Measures  of  Policy  Effectiveness 
(MoPE)  and  Measures  of  Force  Effectiveness  (MoFE); 

•  Required  or  desired  performance  thresholds  and  nominal  Measures  of  Merit  (MoM)  values;  and 

•  The  impact  of  an  improved  volume,  accuracy,  and/or  quality  of  information  on  the  final  outcome. 

7.5  DEVELOPING  AND  SPECIFYING  SCENARIOS 
7.5.1  Prerequisites  in  Scenario  Definition 

Several  prerequisites  are  essential  before  using  scenarios  for  C2  analysis: 

•  Approval:  the  analyst  should  strive  for  the  creation  of  a  family  of  approved  scenarios.  In  creating 
a  family  of  approved  scenarios,  which  reflect  the  mission  objectives  and  force  capabilities  and 
cover  all  significant  warfare  areas,  the  analyst  facilitates  the  scenario  development  process  to  a 
great  extent,  because  references  to  basic  assumptions  and  conditions  can  be  made.  This  will  also 
increase  the  validity  of  the  analysis  in  the  eyes  of  the  client  and  facilitate  comparison  of  the  results 
from  different  studies  and  analyses  that  use  the  same  approved  scenarios; 

•  Breadth:  a  scenario  should  reflect  those  factors  that  are  hypothesised  to  have  a  significant  impact 
on  C2  issues; 

•  Capability:  a  scenario  should  stress  C2  capabilities,  including  human  and  organisational  factors 
(military  and/or  civilian)  where  appropriate;  and 

•  Credibility:  scenarios  should  include  logical  assumptions  about  the  environment  under  analysis. 

Scenarios  should  represent  plausible  real  world  situations.  The  synthetic  scenario  environment  should  be 
consistent  across  OA  studies.  The  scenarios  will  gain  credibility  if  a  broadly  based  scenario  team  is 
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involved  in  the  process  from  the  outset.  This  team  should  include  a  variety  of  perspectives  and  expertise, 
such  as: 


•  OA  analysts  for  defining  the  required  scenario  information,  to  avoid  biasing  the  analysis  by 
selecting  an  inappropriate  scenario  space,  and  for  working  out  the  process  of  framing  of  the 
scenario  space; 

•  Defence  concept  planners  to  propose  options  and  highlight  the  critical  factors; 

•  Policy  makers  to  ensure  that  the  strategic  decision  points  and  the  alternative  options  for  each  of 
these  decision  points  are  clear  and  consistent  with  Defence  Policy;  and 

•  Subject  matter  experts  to  credibly  explore  the  range  of  possibilities  (scenario  space)  and  foster 
discussions. 

C2  Organisation  Infrastructure  and  Operating  Environment 

Organisation  infrastructure  and  environment  are  often  pre-set  conditions  and  not  the  subject  of  the  study. 
They  include  C2  concepts  of  operation,  decision  hierarchy  of  the  units  under  consideration,  degree  of 
technological  competence  relative  to  that  of  the  adversary,  requirements  or  objectives  placed  upon  the 
system  in  terms  of  speed,  accuracy,  flexibility,  etc.,  and  the  impacts  of  terrain,  weather,  and  adversary 
activities. 

C2  Processes 

The  scenarios  need  to  provide  for  the  realistic  execution  of  the  C2  processes.  These  include  the  span  of 
control  of  the  various  military  command  levels  and  civilian  authorities,  information  management  schemes, 
information  flows,  the  elements  of  the  decision  cycle,  the  decision  processes  (course  of  action 
development,  planning,  directing),  and  the  communications  processes  and  capabilities  (data  update  rates, 
throughput,  reliability,  accuracy,  etc.).  All  too  often  these  issues  are  characterised  by  simple  performance 
indicators  and  not  examined  in  detail.  They  may  also  be  an  important  subject  of  the  study.  These  factors 
need  to  be  explicitly  built  into  the  scenarios. 

C2  Systems 

Characteristics  of  C2  systems  are  directly  related  to  system  improvements.  They  include  system 
performance  parameters,  command  and  control  information  systems  (CCIS),  data  availability,  intelligence 
functions  (fusion,  correlation,  aggregation,  etc.),  surveillance,  targeting  and  acquisition  (STA), 
communications  systems,  throughput,  and  so  forth. 

Human  Factors 

C2  studies  are  complex  in  nature.  One  of  the  complicating  factors  is  the  involvement  of  human  beings  and 
their  interpretation  of  a  situation,  order,  or  rule  of  engagement.  These  factors  can  be  covered  by  the 
“aggregation/de-aggregation”  phenomena  in  the  command  chain.  Human  factors  have  to  be  included  in 
the  analysis  and  modelling  activities,  but  guidelines  on  how  to  integrate  the  “human  in  the  loop”  are  partly 
defined  in  the  scenarios  (Chapter  6  -  Human  Decisionmaking  discusses  these  in  some  detail). 

Miscellaneous 

C2  studies  usually  cover  various  levels  of  hierarchy.  However,  the  nature  of  C2  issues  does  not  materially 
change  for  the  various  command  levels.  The  analyst  may  need  to  perform  a  cost/benefit  analysis  on  the 
inclusion  of  lower  level  C2  issues  in  a  closed  simulation  model.  The  analyst  needs  to  consider  to  what 
extent  these  issues  (e.g.  performance  of  a  logistic  information  system)  need  to  be  converted  to  enabling 
factors  (e.g.  sustainability,  operational  delays)  at  the  higher  levels. 
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These  issues  are  mentioned  to  illustrate  that  in  the  scenario  definition  a  great  deal  of  attention  is  required 
to  ensure  that  the  scenario  enables  the  proper  C2  issue  to  be  addressed  in  problem  definition.  The  elements 
are  often  dependent  on  each  other.  For  example,  some  shortcomings  in  C2  systems  can  be  compensated 
for  by  alterations  in  C2  processes.  Similarly,  inefficiencies  in  C2  processes  can  be  met  by  an  adaptable  C2 
organisation.  The  relationships  between  these  issues  should  be  recognised  and  taken  into  account  in  the 
scenarios. 

7,5,2  Approach  to  Scenario  Development 

This  section  describes  a  framework  for  the  definition  of  a  scenario.  Then,  based  on  this  framework, 
some  specific  aspects  of  actually  using  scenarios  for  C2  analysis  are  addressed. 

Scenario  Structure 

The  general  scenario  framework  developed  from  the  NATO  Panel  7  Ad  Floe  Working  Group  on  the 
Impact  of  C3I  on  the  Battlefield  has  been  adopted  (Figure  7.1). 


External 

Factors 

Environment 

Mission  Objectives 

Mission 

Economic/Military/ 

Political/Social/ 

Historic  Situation 

Mission  Constraints  & 
Limitations  ROE 

Military  Scope 

Intensity  \  Tasks 

Joint/Command  ’ 

National  Security  Interests 

Capabilities 
of  Actors 

•  Organisation,  Order  of  Battle,  02,  Doctrine,  Resources,  Lessons  Learned 

•  Weapons 

•  Logistics 

•  Skills 

•  Morale 

Friendly  Forces  Adversary  Forces  Neutral  Forces  Non-Combatants 

Environment 

•  Geography/Region/Terrain 

•  ClimateA/Veather 

•  (Civil)  Infrastructure  (e.g..  Transportation,  Telecommunications,  Energy) 

Figure  7.1:  The  Scenario  Framework. 


At  the  first  two  levels,  a  description  of  the  external  factors  and  the  capabilities  of  the  actors,  including 
national  security  interests;  the  political,  historical,  and  military  situation;  and  the  acting  assumptions, 
boundary  conditions,  and  limitations  related  to  adversaries,  threats,  risks,  coalition  partners,  warfare 
domains  etc.  are  given.  Very  often  a  reference  to  an  approved  scenario  will  suffice. 

At  the  third  level,  the  mission  environment  is  defined.  Whether  it  is  a  generic,  virtual  geographic 
environment  or  a  specific  geographical  area  is  not  important.  What  is  essential  that  the  mission 
environment  be  addressed. 

The  intermediate  level  is  the  most  challenging  one.  The  actual  military  problem  has  to  be  projected  on  the 
mission,  the  military  forces  and  capabilities,  the  civilian  capabilities,  and  the  resources  available. 
A  scenario  must  be  developed  by  coherently  aggregating  a  number  of  components  or  dimensions  with 
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their  attached  values,  taking  into  account  of  the  problem  formulation.  It  should  address  at  least  these 
components: 

•  Geopolitical  situation,  including  historical  aspects; 

•  Geographical  area; 

•  Availability/usability  of  civil  infrastructure; 

•  Terrain  and  climate; 

•  Political/economical/military  objectives; 

•  Level  of  violence,  type  of  warfare  areas,  and  preparation  times; 

•  Mission  context  and  objectives; 

•  Mission  tasks  and  goals; 

•  National  contributions  and  roles  within  the  coalition; 

•  Order  of  battle; 

•  Doctrines,  procedures,  (range  of  acceptable)  rules  of  engagement,  and  concepts  of  operation; 

•  Temporal  factors  (e.g.  anticipated  duration  of  operations); 

•  Desired  end  states; 

•  Opposition/threat/risks; 

•  Adversary  forces  and  their  organisation; 

•  Other  actors  (neutral  and  uncommitted  forces,  non-combatants,  refugees,  lOs,  NGO’s, 
Red  Cross,  etc.); 

•  Level  of  threat  and  risk; 

•  C2  structures; 

•  Interaction  between  friendly,  adversary,  and  other  information  systems;  and 

•  Assumptions/hypotheses/axioms  about  level  of  technology,  impact  on  information  defence, 
etc. 

The  actual  military  problem  will  be  placed  in  the  context  of  the  friendly  and  adversary  military  forces 
involved,  e.g.: 

•  Force  organisation,  C2  structure,  force  components; 

•  Doctrines,  tactics,  rules  of  engagements; 

•  Courses  of  action; 

•  Information  systems;  and 

•  Logistics. 


The  framing  of  scenarios  begins  with  the  identification  of  the  key  dimensions  relative  to  the  problem 
being  addressed.  The  search  of  these  dimensions  is  delicate  and  requires  thorough  reflection,  for  instance, 
on  the  results  of  a  structural  analysis.  Once  identified,  each  key  dimension  or  factor  must  be  characterised 
by  a  range  of  possible  values  or  sectors,  which  enables  the  set  up  of  the  sector-factor  matrix.  This  sector 
factor  matrix  is  then  analysed  through  a  morphological  analysis  (construction  and  reduction  of  the 
morphological  space  or  space  of  “all  possible  scenarios”),  which  provides  the  user  with  a  set  of 
appropriate  scenarios.  The  scenarios  are  dependent  upon  the  problem  and  the  objective  of  the  study  and 
can  range  from  generic  to  very  specific.  The  analyst  will  select  the  level  of  detail  required  to  drive  and 
focus  the  model. 
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7.6  USING  SCENARIOS  IN  C2  ASSESSMENT 

It  has  already  been  noted  that  a  well-formulated  OA  problem  definition,  guidelines  and  directives  for  how 
to  approach  the  analysis  should  accompany  a  scenario.  Emphasis  in  this  COBP  is  given  to  C2  elements  in 
relation  to: 

•  Mission  scope:  as  stated  before,  one  of  the  characteristics  of  C2  is  that  it  can  not  be  studied  in 
isolation.  C2  is  an  integrating  and  enabling  factor.  As  a  consequence  there  is  a  tendency  to 
consider  the  entire  mission,  covering  all  the  arenas  connected  to  it  (from  logistics  to  manoeuvre, 
from  artillery  support  to  close  combat,  from  security/police  issues  to  refugee  management). 
Therefore,  the  typical  mission  scope  for  C2  analysis  will  be  broad; 

•  Levels  of  hierarchy:  the  C2  chain  is  not  limited  to  a  special  hierarchical  level;  information  and 
command  flows  are  running  from  the  lowest  levels  to  the  higher  echelons  and  vice  versa.  As  a 
consequence,  there  is  a  tendency  to  cover  a  wide  set  of  hierarchy  levels  in  considering  a  C2 
problem.  Single  layer  analyses  do  not  represent  the  dynamics  of  military  problems  adequately  to 
answer  most  C2  issues;  and 

•  Aggregation/disaggregation:  the  aggregation  of  data  flows,  data  fusion  in  support  of  intelligence 
processes,  merging  C2  items  to  more  abstract  levels,  etc.  is  difficult  but  manageable.  The  process 
of  deleting,  merging,  and  combining  information  is  reasonably  well  understood.  The  integration 
of  soft  factors  (e.g.  human  and  organisational)  is  less  well  understood  and  makes  the  problem 
more  difficult  to  simulate,  study,  or  analyse.  In  particular,  the  effects  of  interactions  between 
functional  groups  and  echelons  must  be  considered  when  decomposition  is  undertaken  for  analytic 
purposes. 

7,6,1  Mission  Scope  versus  Levels  of  Hierarchy 

Experience  with  prior  modelling  efforts  indicates  that  the  scenario  developer  needs  to  understand  the 
relationships  between  the  scope  of  the  mission,  the  hierarchy  assigned  to  conduct  the  mission,  and  the 
level  of  detail  at  which  the  echelon  operates.  In  general,  as  shown  in  Figure  7.2,  the  broader  the  mission 
scope,  the  higher  the  echelon  required  to  conduct  the  mission.  Also,  higher  command  levels  tend  to  use 
information  at  higher  levels  of  aggregation,  or  less  detail,  than  lower  levels.  The  scenario  developer  should 
attempt  to  operate  in  the  shaded  area  of  the  diagram,  matching  the  echelon  levels  to  the  proper  aggregation 
level  and  mission  scope. 


Figure  7.2:  Intersections  in  Hierarchy. 
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In  many  complex  problems,  the  analyst  is  required  to  subdivide  the  problem  into  smaller  parts,  perhaps 
even  using  different  models  or  model  federations  to  represent  different  levels.  The  scenario  developer 
should  work  in  conjunction  with  the  analyst  to  ensure  that  the  hierarchical  intersections  and  interactions 
(e.g.  organisation  and  infrastructure,  C2  processes,  and  C2  resources)  are  properly  represented,  that  the 
models  are  consistent,  and  their  inputs  and  outputs  are  properly  linked.  Figure  7.3  graphically  depicts  this 
relationship,  with  the  rectangles  X,  Y,  and  Z  representing  different  models. 
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Figure  7.3:  Segmentation  Hierarchy  Range. 


Aggregation/De- Aggregation:  Non-Causality  between  C2  Issues  at  Different  Levels 

Analysis  of  C2  issues  generally  requires  assessing  the  effects  of  events  and  actions  across  command  levels 
to  determine  causality  between  actions  at  one  echelon  and  events  at  another.  For  example,  the  analyst  will 
want  to  know  if  the  capability  to  make  faster  decisions  at  battalion  level  has  an  impact  at  brigade  level  or 
higher.  Some  C2  items  have  an  impact  throughout  the  entire  hierarchical  range,  some  affect  only  one  other 
level,  and  some  are  purely  local,  i.e.,  level-specific.  The  scenario  developer  will  need  to  be  aware  of  the 
analyst’s  requirements  in  this  area  in  order  to  design  the  proper  linkages  between  events. 

7,6,2  Additional  Aspects 

Additional  Areas  of  Consideration: 

•  Specification:  the  purpose  of  the  C2  analysis  will  influence  the  kind  of  scenarios  to  be  used. 
There  are  no  universal  generic  scenarios  for  C2  analysis.  Some  nations  have  “validated”  scenarios 
that  can  be  taken  off  the  shelf  and  modified  to  support  particular  analyses; 

•  Merging  mission  operations  areas:  mission  areas  for  the  various  levels  of  intensity  of  conflict  and 
various  level  of  civilians  involvement  are  not  discrete,  and  it  may  be  necessary  to  include 
elements  of  more  than  one  type  into  a  scenario; 

•  Decomposition:  sometimes  it  might  be  useful  to  decompose  a  scenario  into  two  or  more  detailed 
scenarios  that  each  deal  with  a  certain  subset  of  C2  issues.  This  is  more  or  less  analogous  to  the 
decomposition  into  one  or  more  mission  operations  areas.  It  may  be  necessary  to  add  some 
vignettes  to  look  in  greater  detail  such  that  all  C2  issues  of  interests  are  covered  and  that  the 
whole  range  of  relevant  OOTW  is  explored; 
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•  Adversary-friendly  interaction:  the  attention  given  to  adversary  and  friendly  C2  processes  should 
be  balanced  in  those  cases  where  adversary  activities  are  germane  to  the  problem,  especially  if 
counter  C2  (information  defence)  is  part  of  the  analytic  focus; 

•  Assumptions  and  guidelines:  the  scenarios  are  part  of  the  process  in  developing  an  analysis 
(See  Figure  2.3  in  Chapter  2).  In  the  scenario  phase,  all  scenario  assumptions,  guidelines, 
and  boundaries  for  the  study  should  be  revisited.  Each  study  is  executed  within  the  framework  of 
the  scenario,  and  therefore  the  study  findings  are  only  valid  within  the  limitations  of  the  various 
assumptions  and  artificial  constraints  of  the  scenario; 

•  Traceability:  analysts  should  understand  which  scenario  assumptions  and/or  boundary  conditions 
are  driving  factors  in  the  analysis.  A  detailed  description  of  past  use  of  scenarios  should  be 
maintained  on  a  national  level  in  order  to  avoid  duplication.  Such  a  repository  should  also  contain 
Verification,  Validation,  and  Accreditation  (VV&A)  information  on  the  scenarios;  and 

•  Awareness:  create  awareness  of  the  robustness  of  the  overall  conclusions  and  decisions  and  be 
aware  of  the  degrees  of  uncertainty  in  the  scenario. 


7.7  CONCLUSIONS 

•  Using  a  scenario  for  C2  analysis  is  only  one  part  of  a  larger  analytical  methodology.  The  context 
provided  by  the  scenario  impacts  in  other  areas,  and  the  scenario  in  turn  is  affected  by  those  same 
areas. 

•  Six  prerequisites  should  be  in  place  before  using  a  scenario  for  C2  analysis: 

•  It  should  be  approved  for  the  assessment; 

•  It  should  reflect  the  factors  that  have  significant  impact  on  C2  needs; 

•  It  should  stress  C2  issues; 

•  It  should  be  militarily  credible; 

•  It  should  be  credible  in  terms  of  civil-military  objectives;  and 

•  It  should  facilitate  the  design  process. 

•  At  least  three  C2  elements  should  be  reflected  in  a  scenario  in  order  to  make  it  useful  for  C2  analysis: 

•  The  C2  organisation  and  infrastructure,  including  human  issues; 

•  The  C2  processes;  and 

•  The  C2  systems. 

•  Scenario  guiding  directives  should  indicate  how  the  scenario  has  been  used  in  a  hierarchy  of  scenarios 
(interpretation  of  input  and  output  events,  etc.). 

•  The  actual  C2  analysis  problem  usually  will  be  broader  in  scope  than  OA  will  allow.  Hence  Scenario 
Analysis  in  combination  with  military  and  civilian  judgement  (including  lessons  learned)  must  bridge 
this  gap. 

•  Analysts  need  to  use  multiple  scenarios;  no  single  scenario  is  sufficient. 
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7.8  RECOMMENDATIONS 

Practice: 

•  Organise  a  set  of  scenarios  and  vignettes  that  allow  the  analysis  to  cover  or  sample  the  interesting 
problem  space  for  the  C2  analysis; 

•  Create  a  (national)  base  of  approved  scenarios  and  vignettes  reflecting  the  civil-military  objectives 
within  the  national  hierarchy  of  operations  and  thus  the  required  spectrum  of  military  missions 
including  OOTW  capabilities; 

•  Explicitly  identify  and  describe  the  scenarios  prior  to  the  execution  of  a  study.  However,  it  might 
be  necessary  to  revisit  the  scenario  definition  during  the  conduct  of  the  study; 

•  Information  and  hypotheses  on  threats,  adversary  forces,  and  non-combatants  should  be  addressed 
in  the  scenario; 

•  Explicitly  identify  the  C2  aspects  under  consideration  within  the  problem  definition;  and 

•  During  the  analysis,  the  key  scenario  assumptions  should  be  identified  and  documented. 

Challenges: 

•  Standards  for  judging  the  applicability  and  accreditation  of  (existing)  models  should  be 
developed;  and 

•  For  coalition  C2  assessments  the  scenarios  should  be  developed  or  adapted  by  teams  including 
representatives  from  all  participating  nations. 


7.9  CHAPTER  7  ACRONYMS 

C2 

Command  and  Control 

CCIS 

Command  and  Control  Information  Systems 

COBP 

Code  of  Best  Practice 

10 

International  Organisation 

MoFE 

Measures  of  Force  Effectiveness 

MoM 

Measures  of  Merit 

MoPE 

Measures  of  Policy  Effectiveness 

NGO 

Non-Govemment  Organisation 

OA 

Operational  Analysis 

STA 

Surveillance  Targeting  and  Acquisition 

VV&A 

Verification,  Validation,  and  Accreditation 
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“The  reasonable  course  of  action  in  any  use  of  arms  starts  with  calculation.  Before  fighting,  first  assess 
the  relative  sagacity  of  the  military  leadership,  the  relative  strength  of  the  enemy,  the  size  of  the  armies, 
the  lie  of  the  land,  and  the  adequacy  of  provisions.  If  you  send  troops  out  only  after  making  these 
calculations,  you  will  never  fail  to  win.  ”  -Liu  Ji,  1310-1375,  Lessons  of  War 

“War  is  essentially  a  calculation  of probabilities.  ”  -Napoleon 

“If  the  only  tool  that  you  have  is  a  hammer,  you  tend  to  see  every  problem  as  a  nail.  ”  -attributed  to  a 
Harvard  anthropologist 

TOOLS,  MODELS,  AND  THEIR  APPLICATIONS 

The  purpose  of  this  ehapter  is  to  eonsider  the  best  methods  for  representing  Command  and  Control  (C2) 
systems,  proeesses,  organisations,  and  their  interaction  in  order  to  support  assessments  of  C2  over  the  full 
spectrum  of  operations,  to  include  Operations  Other  Than  War  (OOTW).  The  key  objective  is  to  establish 
an  ‘audit  trail’  from  data  or  information  collected,  through  its  processing,  presentation,  dissemination, 
and  use  to  the  performance  of  C2  processes  and  organisation  as  well  as  to  high-level  measures  of  their 
effects  on  battle  or  operations  outcome. 

This  chapter  of  the  revised  COBP  extends  the  code  to  cover  method  and  tool  considerations  regarding 
OOTW.  The  material  in  this  chapter  is  a  distillation  of  the  best  approaches  and  ideas  being  considered  in 
current  NATO  research  for  representing  C2  across  the  full  spectrum  of  operations  in  all  models/tools. 
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8.1  TYPES  OF  METHODS  AND  TOOLS 

This  chapter  covers  all  tools  (simulations  or  other  quantitative  or  qualitative  techniques),  whether  used 
for  analysis,  training,  or  operational  purposes  that  can  be  used  to  assess  C2  processes,  performance, 
and  effectiveness.  Available  methods  and  tools  can  be  categorised  into  four  distinct  groups: 

•  Data  collection/generation:  methods  and  tools  used  to  either  collect  or  generate  subjective  or 
objective  data  for  subsequent  analysis  from  live,  virtual,  or  constructive  sources,  whether  past, 
present,  or  future; 

•  Data  organisation/relationship:  methods  and  tools  used  to  organise  data  in  some  logical  way, 
or  used  to  establish  relationships  between  data.  These  methods  and  tools,  rather  than  providing  a 
mathematical  solution  to  problems,  tend  to  be  more  qualitative,  subjective,  and  exploratory 
techniques  based  on  expert  opinion,  judgement,  and  interaction,  whether  obtained  directly  or 
through  role  playing.  Although,  in  some  cases,  these  tools/techniques  may  totally  solve  the 
problem  at  hand,  more  often  they  will  illuminate  other  associated  or  sub-problems,  determine 
areas  for  further  analysis,  or  provide  expert  input  “data”  for  more  quantitative  “solving”  tools; 

•  Solving:  methods  and  tools  which  have  been  typically  associated  with  operations  research, 
business,  mathematics,  computer  science,  information  science,  engineering,  or  management 
science  which  tend  to  be  quantitative  in  nature  and  which  usually  consist  of  techniques  providing 
mathematically  derived  solutions,  even  if  the  data  analysed  is  subjective  in  nature;  and 

•  Support:  methods  and  tools  used  to  collect,  organise,  store,  and  explore  typically  large  sets  of 
empirical  data. 

Table  8.1  provides  some  recommended  methods  and  tools  that  fall  into  each  category.  The  intent  here  is 
not  to  provide  an  exhaustive  list  or  to  debate  into  which  category  a  specific  method  or  tool  should  be 
included,  but  to  give  the  range  of  methods  and  tools  available  to  the  analyst  for  C2  assessment.  While  the 
range  and  scope  of  potential  methods  and  tools  is  broad,  clearly  the  emphasis  of  this  chapter  of  the  COBP, 
and  of  the  analysis  community  at  large  over  recent  years,  is  on  constructive  modelling  and  how  best  to 
enhance,  orchestrate,  and  apply  it  to  the  assessment  of  C2  impacts  on  battle/operations  outcome. 


Table  8.1:  Some  Recommended  Tools  by  Category  Type 


Data 

Generation 

Data 

Collection 

Data  Organisation/ 
Relationship 

“Solving” 

Support 

Real  World  Operations 

• 

After  Action  Reviews 

• 

• 

Historical  Analysis 

• 

• 

Expert  Elicitation 

• 

Constructive  Simulations 

• 

• 

• 

Virtual  Simulations 

• 

• 

• 

Exercises 

• 

• 

Experiments/ 

Experimental  Design 

• 

• 

Game  Theory 

• 

• 

Army  C2  Evaluation  System  (ACCES) 

• 

• 

Headquarters  Effectiveness  Assessment 
Tool  (HEAT) 

• 

• 

Causal  Mapping 

• 

Multi-Criteria  Decision  Analysis  (MCDA) 

• 

• 

Regression  Analysis 

• 

• 

Factor  Analysis 

• 

• 

Bayesian  Networks 

• 

• 
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Table  8.1:  Some  Recommended  Tools  by  Category  Type  (cont’d) 


Data 

Generation 

Data 

Collection 

Data  Organisation/ 
Relationship 

“Solving” 

Support 

Neural  Networks 

• 

• 

Systematic  Approaches 

• 

• 

Mathematical  Programming 

• 

• 

Heuristic  Search 

• 

Genetic  Algorithms 

• 

• 

Project  Management  Tools 

• 

Data  Analysis 

• 

Geographical  Information  Systems 

• 

Visualisation 

• 

• 

Databases 

• 

• 

Checklists 

• 

Spreadsheets 

• 

8.2  ISSUES 

The  NATO  analytic  community  has  had  many  challenges  in  the  past  analysing  the  effectiveness  of 
C2  related  systems  and  determining  what  it  is  that  sets  it  apart  from  other  types  of  operational  analysis. 
The  added  complexity  and  number  of  confounded  variables  in  OOTW  make  analysis  of  these  operations 
even  more  challenging.  The  key  to  the  problem,  no  matter  where  on  the  spectrum  of  conflict  the  analysis 
is  located,  lies  in  making  a  properly  quantified  linkage  between  C2  Measures  of  Performance  (MoP), 
such  as  communication  system  delays,  C2  Measures  of  Measures  of  C2  Effectiveness  (MoCE),  such  as 
planning  time,  and  their  resultant  impact  on  higher  level  Measures  of  Force  Effectiveness  (MoFE) 
or  Measures  of  Policy  Effectiveness  (MoPE),  which  capture  the  effects  on  battle  or  operations  outcome. 
These  higher  level  MoFE/MoPE  are  required  in  order  to  be  able  to  trade  off  investment  in  C2 
systems  against  investment  in  combat  systems  such  as  tanks  or  aircraft.  At  present,  there  is  no  routine 
way  of  making  this  linkage,  nor  any  one  tool  that  can  be  applied  to  generate  the  required  measures. 
Hence,  all  analyses  of  C2  issues  demand  a  high  level  of  creative  problem  structuring  and  approach, 
and  selection  and  application  of  a  range  of  available  analysis  tools,  to  overcome  this  challenge. 

The  range  of  issues  is  very  broad  and  challenging.  Particularly  challenging  are  the  following  issues  that 
will  be  addressed  in  more  detail  later  in  this  chapter: 

•  Representation  of  human  behaviour  (e.g.  rule-based,  algorithmic,  or  “human-in-the-loop”); 

•  Homogeneous  models  versus  hierarchies/federations; 

•  Stochastic  versus  deterministic  models; 

•  Adversarial  representation; 

•  Verification,  Validation,  and  Accreditation  (VV&A);  and 

•  The  conduct  of  sensitivity  analysis  and  other  ways  of  dealing  with  uncertainty. 

Of  particular  importance  for  analysis  of  OOTW : 

•  Selecting  an  orchestrated  set  of  tools  that  generate  the  required  MoM; 

•  Scoping  the  analysis  considering  tool  availability; 

•  Considering  the  human  dimension  in  tool  selection  early  in  the  process;  and 

•  Ensuring  tools  selected  have  the  trust  and  confidence  of  the  decisionmaker(s). 
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8.2.1  Representation  of  Human  Behaviour:  Rule-Based,  Algorithmic, 
or  “Human-in-the-Loop” 

In  developing  methods  and  tools  that  represent  the  process  and  performance  of  C2  explicitly, 
most  approaches  until  very  recently  have  been  founded  on  the  artificial  intelligence  (AI)  methods  of  expert 
systems.  These  represent  the  commander’s  decisionmaking  process  (at  any  given  level  of  command)  by  a 
set  of  interacting  decision  rules.  The  advantage  of  such  an  approach  is  that  it  is  based  on  sound 
AI  principles.  However,  in  practice  it  leads  to  tools  which  are  large,  complex,  and  slow.  The  decision 
rules  themselves  are,  in  many  cases,  very  scenario  dependent  and,  as  noted  in  Chapter  6  -  Human 
Decisionmaking,  human  factors  and  organisational  expertise  may  be  needed  on  a  project  team  to  treat 
these  issues  correctly. 

These  factors  were  not  a  problem  when  the  Cold  War  prevailed.  There  was  sufficient  time  to  complete 
extended  analyses,  and  one  key  scenario  dominated.  However,  in  the  post-Cold  War  environment, 
such  certainties  have  evaporated.  Indeed,  uncertainty  is  now  one  of  the  key  drivers  of  analysis.  There  is  an 
increasing  requirement  to  consider  large  numbers  of  scenarios  and  to  perform  a  wide  range  of  sensitivity 
analyses.  This  has  led  to  a  requirement  for  ‘lightweight,’  fast  running  tools,  that  can  easily  explore  a  wide 
range  of  scenarios,  yet  still  appropriately  represent  C2.  Some  have  begun  to  explore  advanced  algorithmic 
tools  based  on  Bayesian  mathematics,  catastrophe  theory,  and  complexity  theory.  Such  approaches  to  the 
representation  of  C2  are  at  the  core  of  a  new  generation  of  closed  form  constructive  simulation  models  that 
are  beginning  to  be  used  for  analysis  (Moffat,  2002;  Moffat,  2000). 

Many  analyses  employ  “human-in-the-loop”  techniques  in  order  to  ensure  realistic  human  performance  or 
to  check  assumptions  and  parameters.  However,  “human-in-the-loop”  techniques  are  expensive  and 
require  the  inclusion  of  soft  factors  and  their  attendant  MoM.  The  introduction  of  “human-in-the  loop” 
introduces  a  source  of  variances  and  uncertainty.  The  increased  cost,  complexity,  and  uncertainty  of 
“human-in-the-loop”  methods  often  requires  analysts  to  limit  the  use  of  these  techniques  rather  than 
employ  them  as  the  primary  analytical  method. 

8.2.2  Homogeneous  Model-Tools  versus  Hierarchies/Federations 

In  order  to  build  an  “audit  trail”  that  traces  the  interrelationships  among  individual  C2  systems,  processes, 
and  organisations,  as  well  as  their  impacts  on  mission  operational  outcomes,  there  is  a  need  to  represent 
the  key  detailed  processes  involved,  such  as  the  transmission  of  communications  across  the  battlefield  and 
the  impact  of  logistics  on  decisionmaking.  Taking  this  as  an  example,  the  question  then  arises  as  to 
whether  all  the  transmission  media  (radio,  satellites,  etc.),  with  their  capacities,  security  level, 
communications  protocols,  etc.,  should  be  represented  explicitly,  or  whether  these  details  should  be  split 
out  in  a  supporting  model.  Similarly,  the  details  of  logistics  could  be  undertaken  as  part  of  the  main  model 
or  in  a  specialised  supporting  model.  Supporting  models  could  be  run  off-line,  providing  sets  of  input  data 
to  the  main  model  (giving  rise  to  a  model  hierarchy)  or  they  could  be  run  in  real  time  interaction  with  the 
main  model  (giving  rise  to  a  model  federation).  In  the  off-line  mode,  the  main  model  would  generate 
demands  on  the  communications  and  logistics  systems.  The  supporting  models  would  check  if  these 
demands  could  be  satisfied.  If  not,  communication  delays  and  logistics  constraints  in  the  main  model 
would  be  increased,  and  the  main  model  re-run.  This  would  have  to  be  done  a  number  of  times  to  bring  the 
main  and  supporting  models  into  balance.  However,  such  an  approach  can  generate  valuable  analytical 
insights. 

Figure  8.1  shows  the  main  model-tool  producing  (in  addition  to  its  MoFE)  a  detailed  set  of  dynamic 
demands  on  the  communications  (such  as  capacity  required  of  different  communications  systems  as  a 
function  of  simulated  time),  and  logistics  processes  (demands  for  transport  and  key  consumables),  in  order 
to  achieve  the  assessed  levels  of  MoFE.  These  are  then  fed  back  into  detailed  model-tools  of  the 
communications  and  logistics  infrastructure.  Those  supporting  model-tools  can  then  be  matched  against 
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the  dynamic  demand  placed  on  the  communications  and  logistics  infrastructure  to  the  available  capacity. 
If  there  is  a  mismatch,  the  assumptions  in  the  main  model-tools  are  adjusted  iteratively  to  bring  the  two 
model-tools  into  balance.  This  approach  is  more  flexible  and  reactive  for  a  large  set  of  C2  assessments. 
However,  this  approach  increases  the  complexity  of  the  architecture  (number  of  linked  sub-processes, 
failure  of  the  sub-model,  etc.). 


Linkage 
Primary  Model 
Supporting  Model 


Figure  8.1:  Model  Linkage. 

A  similar  approach  can  be  applied  to  concepts  of  operation.  In  some  models,  it  is  possible  to  describe  a 
concept  of  operations  as  a  sequence  of  standard  missions  (e.g.  attack,  defend,  move).  These  missions  can 
then  be  analysed  to  determine  the  demands  they  place  on  the  supporting  infrastructures.  This  can  be  tested 
off-line  to  see  if  the  infrastructure  can  cope.  Again,  this  would  have  to  be  iterated  a  number  of  times, 
but  provides  an  ability  to  relate,  in  an  understandable  way,  the  infrastructure  capacity  to  its  ability  to 
support  a  defined  concept  of  operations  (and  hence  battle  outcome).  In  addition  to  the  use  of  such 
hierarchies  of  supporting  models  in  an  off-line  mode,  it  is  possible  to  create  real-time  federations  of  such 
models  to  represent,  inter  alia,  combined  or  joint  operations. 

8,2,3  Stochastic  versus  Deterministic  Models 

Stochastic  and  deterministic  models  differ  in  how  they  treat  variables.  Stochastic  models  incorporate  the 
attributes  of  the  probability  density  (or  distribution)  functions  associated  with  model  variable  into  run  time 
calculations.  The  actual  values  of  model  variables  will  therefore  differ  each  time  the  model  is  run. 
Thus,  the  output  of  a  stochastic  model  will  yield  different  results  each  time  even  when  a  set  of  inputs  is 
“fixed”.  Deterministic  models  utilise  point  estimates  for  the  values  of  model  variables,  and  hence  for  any 
given  set  of  inputs  the  model  will  produce  a  single  output. 

Stochastic  and  deterministic  models  each  have  their  advantages.  The  selection  of  which  to  use  or  how  to 
employ  each  type  of  model  in  an  assessment  depends  upon  the  nature  of  the  problem  and  other  elements 
of  the  solution  approach.  Whether  the  model  is  time  step  driven  or  event  sequence  driven  will  also  impact 
on  the  selection  of  the  appropriate  models(s). 

Deterministic  Models 

The  merits  of  a  deterministic  approach  are  that  run-times  are  reduced,  and  there  is  a  single  ‘thread’ 
connecting  the  input  data  and  the  results,  making  the  analysis  of  the  tool  output  potentially  easier. 
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The  justification  for  using  deterministic  (expected  value)  tools  is  usually  based  on  the  assumption  that, 
due  to  the  large  number  of  stochastic  processes  involved  in  combat  on  higher  levels,  their  results  converge 
rather  quickly  to  the  mean  values  obtained  from  stochastic  models. 

Stochastic  Models 

Stochastic  models  do  provide  significantly  more  information  than  deterministic  tools.  They  are  deemed  to 
be  indispensable  for  an  assessment  of  the  robustness  of  the  results  as  well  as  providing  the  “data”  needed 
for  risk  analysis.  These  additional  benefits  come  at  the  cost  of  significantly  higher  run  time  requirements 
and  a  more  complex  analysis  task. 

Chaos  theory  shows  that  structural  variance  (or  ‘deterministic  chaos’)  can  occur  when  sets  of  decision 
rules  interact  in  the  simulation  of  a  dynamic  process.  Small  changes  in  initial  conditions  can  lead  to  very 
different  trajectories  of  system  evolution.  Any  simulation  model  of  combat,  with  a  representation  of  C2, 
has  to  face  this  kind  of  problem.  The  merits  of  a  deterministic  approach  are  that  run  times  are  reduced  and 
there  is  a  single  ‘thread’  connecting  the  input  data  and  the  results,  making  analysis  of  the  model  output 
potentially  easier.  However,  the  representation  of  the  C2  process  (whether  using  decision  rules  or  not) 
gives  rise  to  a  number  of  alternative  decision  options  at  any  given  moment,  and  can  thus  potentially  give 
rise  to  such  ‘deterministic  chaos’.  If  such  effects  are  likely  to  arise,  one  solution  is  to  use  stochastic 
modelling.  The  use  of  stochastic  sampling  in  the  model,  together  with  multiple  replications  of  the  model, 
gives  rise  to  a  distribution  of  outcomes,  which  is  much  more  resistant  to  such  chaotic  effects. 

US  Army  Training  And  Doctrine  Command  (TRADOC)  has  been  experimenting  with  Deterministic 
Combat  Models.  “A  potential  alternative  solution,  when  the  issue  under  study  warrants,  is  to  conduct 
analysis  of  multiple  runs  of  a  deterministic  model  where  the  initial  states  of  information  systems  are 
varied”  (Bailey,  2001). 

8.2.4  Representing  Adversary  Forces 

Historically,  adversary  capabilities  and  behaviours  were  often  fully  scripted  or  heavily  constrained. 
This  was  more  appropriate  in  Cold  War  contexts  than  it  is  today.  However,  it  was  never  ideal  for  C2 
analysis  because  the  dynamic  interaction  among  friendly,  adversary,  and  other  forces  is  a  critical  element 
of  C2  representation.  Today,  much  more  robust  adversary  representation  of  operational  capabilities  and 
choices  are  employed  and  indeed  are  necessary.  Analysts  must  consider  not  only  a  range  of  scenarios, 
but  also  the  range  of  possible  adversary  actions  and  reactions. 

8.2.5  Verification,  Validation,  and  Accreditation  (W&A) 

VV&A  has  historically  been  a  challenge  for  model  development  efforts,  but  is  particularly  challenging  for 
C2  modelling.  This  is  due  to  the  variability  inherent  in  most  C2  processes,  especially  those  that  involve 
the  human  aspects  of  information  processing  and  decisionmaking.  The  approach  to  VV&A  needs  to  be 
carefully  considered,  particularly  in  light  of  the  need  to  assess  future  C2  systems  and  capabilities  in 
association  with  new  concepts  of  operation,  new  organisational  forms,  new  doctrine  and  asymmetrical 
adversaries. 

8.2.6  Selecting  an  Orchestrated  Set  of  Tools 

The  natural  tendency  of  an  analyst  is  to  simplify  a  problem.  Part  of  that  simplification  is  to  select  a  tool, 
preferably  only  one,  which  will  meet  the  analysis  requirements.  In  the  analysis  of  C2  of  combat 
operations,  this  may  be  possible  if  the  analysis  is  properly  scoped.  In  the  analysis  of  OOTW  C2,  the  issues 
are  typically  too  numerous,  the  variables  too  confounding  and  the  scope  too  broad  for  one  tool  to  satisfy 
all  analysis  requirements.  An  orchestrated  set  of  complementary  tools  will  normally  be  required. 
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8.2.7  Scoping  the  Analysis  Considering  Tool  Availability 

An  analyst  must  always  scope  the  analysis  during  problem  formulation  to  enable  it  to  be  accomplished 
within  available  resource  constraints.  During  problem  formulation,  however,  the  consideration  of  available 
tools  has  typically  not  been  a  driving  factor.  Also,  in  the  analysis  of  C2  combat  operations  this  may  not  be 
a  problem  because  selection  of  the  tool(s)  to  be  used  is  more  obvious  based  on  past  experience.  However, 
for  OOTW  C2  analysis,  the  availability  of  tools,  and  their  orchestration,  requires  more  consideration  early 
in  the  process  (i.e.  during  problem  formulation). 

8.2.8  Consideration  of  the  Human  Dimension 

C2,  by  its  very  nature,  is  closely  linked  to  human  behaviour,  and  its  analysis  requires  careful  consideration 
and  inclusion  of  the  human  dimension.  Often,  as  a  way  of  simplifying  the  analysis,  the  C2  assessment 
team  eliminates  these  considerations  by  assuming  that  human  commanders  and  their  staffs  are  not  affected 
by  their  environment  and  will  always  make  the  best  decision,  and  the  same  decision,  given  the  required 
information.  This  is  unsafe  and  not  good  practice.  This  has  been  especially  true  for  the  analysis  of  OOTW 
C2,  even  though  this  analysis  is  often  more  impacted  by  the  human  dimension  than  C2  for  combat 
operations. 

8.2.9  Ensuring  Trust  and  Confidence  in  the  Tools 

Analysts  select  and  apply  the  tools  of  their  trade  based  on  what  those  tools  can  do  for  them  in 
accomplishing  their  analysis  objectives.  Over  time  they  become  comfortable  with  certain  tools  and  the 
customers  for  their  analysis  also  develop  a  trust  in  the  tools  and  confidence  that  they  will  produce  valid 
results  for  them.  For  OOTW  C2  analysis,  given  that  the  tools  are  more  numerous,  must  be  orchestrated  to 
work  together,  and  are  sometimes  unknown  or  not  understood  by  the  customer,  the  development  of  trust 
and  confidence  in  these  tools  is  difficult  to  achieve. 

8,3  NEW  METHODS  AND  EMERGING  PRACTICES 

Given  the  inherent  problems  associated  with  the  issues  described  previously,  a  best  practice  for  the 
application  of  analysis  tools  for  C2  analysis  is  still  emerging.  This  best  practice  is  described  below, 
first  for  tools  in  general,  then  for  models,  both  at  the  model  level  itself  and  at  the  algorithm  level. 

8,3,1  Selection  of  Methods  and  Tools 

The  selection  of  tools  to  apply  to  C2  analysis  should  be  based  both  on  evaluation  of  the  candidate  tools 
themselves  against  a  set  of  evaluation  criteria  and  on  consideration  of  the  type  of  study  to  be  undertaken. 
The  evaluation  criteria  for  tool  selection  includes  criteria  related  to  the  functionality  of  the  candidate  tool 
and  to  the  performance  of  the  candidate  tool.  The  following  are  established  evaluation  criteria  for  tool 
selection.  They  are  as  applicable  to  tool  selection  for  OOTW  C2  as  for  combat  C2  analysis. 

•  Functionality-related  tool  selection  criteria: 

•  Resolution:  the  level  of  detail  in  representation  of  entities  within  the  tool; 

•  Completeness/scope:  the  extent  to  which  the  tool  is  able  to  address  analysis  issues; 

•  Functionality:  the  extent  to  which  the  tool  represents  the  full  range  of  functions; 

•  Explicitness:  the  ability  of  the  tool  to  explicitly  represent  required  entities; 

•  MoM  Generation:  the  ability  of  the  tool  to  generate  the  MoM  required;  and 
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•  VV&A:  the  determination  of  whether  the  tool  has  been  verified,  validated,  and/or  accredited  for  it 
intended  use.  (Note:  Not  all  NATO  member  nations  recognise  the  term  “Accreditation”. 
Accreditation  here  refers  to  some  form  of  formal  approval  to  use  the  model  for  the  analysis 
intended.) 

•  Performance-related  tool  selection  criteria: 

•  Responsiveness:  the  amount  of  time  between  request  and  receipt  of  information; 

•  Simplicity:  the  ease  of  preparation  and  use  the  tools; 

•  Preparation/use  time:  the  length  of  time  necessary  to  prepare  and  use  the  tool; 

•  Data  availability  and  parameters:  the  ease  in  acquiring  or  generating  the  necessary  data  or 
parameters  for  tool  use; 

•  Interoperability:  the  ability  of  the  tool  to  interoperate  with  other  tools; 

•  Resource  requirements:  the  amount  of  resources  (time,  personnel,  and  funds)  required;  and 

•  Credibility:  the  extent  to  which  the  customers  and  users  accept  tool  results. 

8,3,2  Using  Models 

The  potential  for  exploiting  recent  advances  in  mathematics  in  order  to  create  fast  running  model-tools 
was  noted  earlier.  Such  models  have  exploited  emerging  approaches,  such  as  complexity  theory, 
chaos  theory,  catastrophe  theory,  and  game  theory  in  order  to  produce  a  ‘good  enough’  representation. 
They  can  be  used  to  complement  more  complex,  detailed  models  of  the  problem  area.  In  many  cases, 
a  tailoring  of  models,  or  other  tools,  will  be  required  to  properly  address  the  analysis  issues  at  hand 
(Moffat,  2002;  Moffat,  2000). 

Model  Federations 

A  number  of  new  approaches  share  a  key  set  of  characteristics.  First,  an  object-oriented  approach  within 
the  model-tool  allows  different  objects  to  be  brought  together  to  represent  the  complete  command  process, 
rather  like  ‘Lego^’^  bricks.’  Such  a  philosophy  also  encourages  the  development  of  model- tools  based  on 
holistic  and  evolutionary  principles.  In  other  words,  always  capture  a  complete  model  of  the  process, 
including  the  parts  whose  representation  is  still  unclear.  As  understanding  develops,  improve  those  parts 
(or  objects)  which  were  rudimentary  at  the  start.  At  the  next  level  up,  the  use  of  run-time  interfacing 
allows  different  model-tools  to  be  brought  together  to  create  a  federation  to  represent  the  process  under 
study.  This  federation  then  may  also  have  to  be  integrated  with  the  use  of  a  mix  of  tools,  which  include 
techniques  other  than  modelling,  to  fully  address  the  study  issues. 

Agent-Oriented  Modelling 

A  second  key  aspect  is  the  description  and  representation  of  the  C2  process  through  agent  modelling  and 
programming  techniques.  Modelling  of  the  C2  process  as  a  group  of  agents,  based  on  artificial  intelligence 
concepts,  favours  the  capture  of  the  cognitive  nature  of  command  tasks.  Agents  can  be  implemented,  in  an 
object-oriented  environment,  as  either  objects  (e.g.,  actor  or  “applef’  type  of  agents),  or  aggregates  of 
objects  (coarse-grain  agents).  Such  agents  interact  with  each  other  through  a  messaging  infrastructure. 
The  term  “agent-oriented  modelling”  is  suggested  as  a  way  of  capturing  this  idea. 

Linking  of  Performance  Model-Tools  to  Effectiveness  Tools 

This  third  idea,  used  in  a  number  of  NATO  countries,  uses  a  structured  hierarchy  of  model-tools  to  create 
an  audit  trail  from  C2  systems,  processes,  and  organisations  through  to  battle  operations  outcome. 
The  idea  is  to  create  supporting  performance  level  model-tools  of  particular  aspects  of  the  process 


8-8 


RTO-TR-081 


METHODS  AND  TOOLS 


(e.g.,  communications,  logistics)  which  can  be  examined  at  the  performance  level.  These  then  form  inputs 
to  higher  level  force  on  force  models.  This  ensures  that  the  combat  models  themselves  do  not  become 
overly  complex.  We  have  already  discussed  this  in  the  sense  of  hierarchies  of  support  models. 

For  example,  in  Figure  8.2,  a  detailed  model  of  the  intelligence  system  can  be  very  complex,  if  we  wish 
to  take  account  flow  of  intelligence  requirements,  tasking,  collection  processes,  fusion  processes, 
and  intelligence  products.  In  order  to  analyse  the  impact  of  intelligence,  it  is  important  to  have  all  of  this 
detail,  but  it  does  not  necessarily  have  to  be  represented  explicitly  in  the  main  model.  A  supporting  model- 
tool  which  captures  all  of  this  detail  can  be  created  (or  used  if  one  already  exists)  in  order  to  produce 
outputs  at  the  MoCE  level,  such  as  speed  and  quality  of  intelligence.  These  can  then  form  inputs  to  the 
main  simulation  model.  The  main  model-tool  then  takes  these  into  account  in  producing  its  own  outputs. 
These  will  now  be  at  the  MoFE  level.  Examples  are  friendly  casualty  levels,  adversary  attrition,  and  time 
to  achieve  military  objectives. 


Figure  8.2:  Model-Tool  Hierarchy. 


Scanning  Scenario  Space 

The  use  of  very  fast  model-tools  to  scan  the  overall  space  of  possibilities  and  to  identify  areas  of  concern 
for  further  analysis  appears  to  give  a  good  balance  between  the  use  of  simple  and  complex  modelling 
approaches.  Fast  models,  which  are  simpler  but  may  have  less  analytic  depth,  allow  the  analyst  to  scope 
the  problem  and  determine  the  degree  of  complexity  a  model-tool  must  represent  in  order  to  conduct  the 
desired  level  of  analysis. 

Decisionmaking  Process 

It  is  important  to  have  a  proper  representation  of  the  decisionmaking  process  in  order  to  establish  the 
link  from  C2  performance  through  MoE  to  overall  MoFE  and  to  represent  information  operations  (10) 
effects  such  as  Counter  C2  or  digitisation  of  the  battlespace.  Representation  of  the  decisionmaking  process 
itself,  however,  remains  difficult  because  of  the  difficulty  in  representing  human  performance,  command 
styles,  and  organisational  relationships. 

Parameter  Development  Context 

Finally,  it  may  become  necessary  to  generate  new  or  additional  data  to  validate  new  or  existing  model- 
tools  to  incorporate  C2  factors.  This  may  be  especially  true  in  the  case  of  integrating  soft  factors  into 
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C2  analysis.  Possible  methods  include  field  trials,  “model-test-model”,  or  advanced  warfighting 
experiments.  Field  trials  are  used  if  uncertainty  revolves  around  measurable  factors  that  are 
only  observable  in  the  field  or  are  not  reproducible  in  the  laboratory.  Model-Test-Model  (M-T-M) 
or  Model-Exercise-Model  (M-E-M)  is  used  as  part  of  an  iterative  process  to  develop  and  apply 
systematically  more  in-depth  and  sophisticated  model-tools  and,  in  some  cases,  more  simplistic  model- 
tools  to  increase  their  validity.  The  original  model-tool  is  executed,  modified  based  on  results  of  the  test 
or  experiment,  and  executed  again  until  it  has  developed  a  sufficient  representation  of  a  complex 
process.  Experiments,  like  advanced  warfighting  experiments,  are  useful  in  modelling  new,  large  scale, 
and  complex  interactions  for  which  little  data  or  few  validated  tools  exist.  Each  approach  requires 
additional  time  and  resources,  and  the  data  sets  may  not  be  validated  for  some  time. 


8.4  C2  MODELLING  GUIDELINES 

A  number  of  common  ideas  have  emerged  which  are  worth  consideration  for  new  modelling  and  tool 
developments.  Figure  8.3  shows  how  each  of  the  following  common  ideas  are  represented  within  an  ideal 
command  and  control  model: 

•  Understanding  of  adversary  intent  can  be  represented  by  having  a  number  of  prescribed  intents  or 
options,  which  are  updated  in  an  advanced  data  architecture,  or  Bayesian  way,  as  more 
information  becomes  available; 

•  Representing  headquarters  explicitly  in  the  model-tool  allows  proper  representation  of 
Information  Warfare  (IW)  effects  such  as  counter  C2; 

•  Explicit  representation  of  the  “recognised  picture”  within  each  headquarters  (HQ)  allows  the 
model-tool  to  run  based  on  different  perceptions  by  each  individual  unit  on  each  side  represented. 
This  allows  the  effects  of  aspects  such  as  deception,  shock,  and  surprise  to  be  explicitly 
considered; 

•  Represent  information  as  a  commodity.  This  consideration  is  the  most  critical  and  difficult  to 
implement,  but  is  the  foundation  for  the  other  guidelines,  as  well  as  for  the  model  itself 
Information  should  be  considered  as  a  resource  that  can  be  collected,  processed,  and  disseminated. 
It  includes  information  about  adversary,  friendly,  and  other  forces,  considerations  such  as 
political-military  factors  and  rules  of  engagement  (ROE),  as  well  as  environmental  information 
such  as  weather  and  terrain.  Information  should  posses  dynamic  values  such  as  accuracy, 
relevance,  timeliness,  completeness,  and  precision.  These  values  should  in  some  way  affect  other 
activities  within  the  model,  to  include,  when  appropriate,  combat  functions; 

•  Represent  the  realistic  fiow  of  information  throughout  the  operational  environment.  Information 
has  a  specific  source,  and  that  source  is  usually  not  the  end  user  of  the  information.  A  requirement 
exists,  therefore,  to  move  information  from  one  place  to  another  in  the  operational  environment. 
Communications  systems  of  all  forms  exist  to  accomplish  this  movement.  These  systems  can  be 
analogue  or  digital.  Information  can  be  lost  and/or  degraded  as  it  flows  around  the  operational 
environment.  The  model-tool  should  represent  the  communications  systems  and  account  for  these 
degradation  factors  as  it  represents  information  flow; 

•  Represent  the  collection  of  information  from  multiple  sources  and  tasking  of  information 
collection  assets.  This  guideline  applies  equally  to  adversary,  neutral,  and  friendly  and  other  force 
information.  For  the  collection  of  adversary  and  other  force  information,  the  model-tool  should 
represent  a  full  suite  of  sensors  and  information  collection  systems,  and  the  ability  of  these 
systems  to  be  tasked  to  collect  specific  information.  For  the  collection  of  friendly  information, 
this  consideration  is  just  as  critical.  Knowledge  of  one’s  own  capability  in  combat,  as  well  as  that 
of  the  adversary  and  other  forces,  is  essential  for  effective  decisionmaking; 
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•  Represent  the  proeessing  of  information.  Information  is  rarely  valuable  in  its  original  form. 
It  usually  has  to  be  proeessed  in  some  way.  Typieal  proeessing  requirements  inelude  filtering, 
eorrelation,  aggregation,  disaggregation,  and  fusion  of  information.  These  proeesses  ean  be 
aeeomplished  by  either  manual  or  automated  means.  The  ability,  or  inability,  to  properly  proeess 
information,  and  the  time  it  takes  ean  have  a  direet  bearing  on  combat  operational  outcome; 

•  Represent  C2  systems  as  entities  in  the  operational  environment.  C2  systems  perform  information 
collection,  processing,  dissemination,  and  display  functions.  They  should  be  explicitly  represented 
as  entities  that  can  be  targeted,  degraded,  and/or  destroyed  by  either  physical  or  non-physical 
means.  Additionally,  the  model-tool  should  account  for  continuity  of  operations  of  critical 
functions  during  periods  of  system  failure  or  degradation; 

•  Represent  unit  perceptions  built,  updated,  and  validated  from  the  information  available  to  the  unit 
from  its  information  systems.  This  is  a  critical  requirement.  Each  unit  should  have  its  own 
perceptions,  gaining  knowledge  from  superior,  subordinate,  or  adjacent  units  only  when 
appropriate; 

•  Represent  commander’s  decisions  based  on  the  unit’s  perception  of  the  battlefield.  Each  unit 
should  act  based  on  what  it  perceives  the  situation  to  be  informed  by  its  commander’s  intent  of 
mission,  goals,  constraints,  and  biases,  not  based  on  ground  truth  available  within  the  model. 
When  a  unit  takes  action  based  on  inaccurate  perceptions,  it  should  suffer  the  appropriate 
consequences;  and 

•  Represent  10  for  each/all  combatants.  With  information  so  critical  to  combat  operations  outcome, 
the  model-tool  should  be  able  to  represent  the  deliberate  attack  and  protection  of  information, 
information  systems,  and  decisions.  This  applies  to  all  sides  represented  in  the  model. 

As  shown  in  Figure  8.3,  explicit  representation  of  these  information  operations  elements  within  a 
command  and  control  will  facilitate  the  assessment  of  human  factors,  psychological  operations,  deception, 
C2  systems  effectiveness,  and  staff  structure  issues. 
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Figure  8.3:  Modelling  Guidelines. 
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8.5  CONCLUSIONS 

The  following  conclusions  are  made  regarding  the  strengths  and  weaknesses  of  current  C2  tools/modelling 
approaches. 

8.5.1  Strengths  in  Current  C2  Tools/Modelling 

An  assessment  of  current  C2  modelling  approaches  employed  against  the  guidelines  above  show  that, 
while  they  may  not  yet  be  satisfied,  there  are  some  strengths  in  the  C2  tools/modelling  approaches 
currently  being  implemented: 

•  There  is  a  common  understanding  of  issues.  This  has  not  always  been  the  case.  With  the  inherent 
complexity  of  C2,  as  well  as  the  challenges  in  modelling  such  a  complex  subject,  there  has  been  a 
tendency  in  the  past  to  ignore  the  subject,  or  just  to  accept  it  as  something  that  is  too  complex  to 
address.  This  does  not  seem  to  be  the  case  now.  Perhaps  through  the  emergence  of  new 
technologies  and  through  the  work  of  groups  such  as  the  NATO  SAS  groups,  the  analysis  and 
modelling  of  C2  is  now  considered  possible.  Member  nations  now  seem  to  have  both  an 
understanding  of  C2  and  its  importance  to  combat  and  OOTW  operations  and  a  common 
understanding  of  the  modelling  challenges  that  exist; 

•  There  is  wide  application  of  C2  modelling.  Member  nations  now  apply  C2  modelling  and  analysis 
to  a  wide  range  of  issues.  These  issues  include  those  associated  with  investment,  requirement 
identification,  force  structuring  and  operational  support.  In  all  of  these  areas,  there  is 
understanding  of  the  sensitivity  and  criticality  of  C2  to  the  proper  analysis  of  combat  operations 
outcome.  Selection  of  the  model-tools  to  apply  to  a  particular  problem  should  be  based  on 
evaluation  of  specific  criteria,  as  discussed  previously  in  this  chapter; 

•  Although  each  nation  develops  its  models  for  different  purposes  and  tailors  their  models  and  other 
tools  for  specific  issues,  there  exists  a  commonality  of  approaches  in  different  nations  that  serves 
to  strengthen  their  collective  merits.  These  common  approaches  are  an  outgrowth  of  the  modelling 
technologies  now  available,  but  also  result  from  shared  experiences  by  member  nations; 

•  Most  of  the  progress  and  success  in  C2  modelling  has  been  with  regard  to  high-intensity  combat. 
This  is  perhaps  due  to  the  belief  by  many  that  a  high-intensity  combat  scenario  is  still  most 
appropriate  for  the  analysis  of  combat,  particularly  for  analysis  of  primary  combat  systems. 
Progress,  therefore,  has  been  focused  on  embellishing  high- intensity  combat  analysis  with  C2 
improvements  to  models.  Unfortunately,  low-intensity  combat  and  OOTW  modelling  and  analysis 
have  not  received  the  same  level  of  attention  until  recently; 

•  There  is  wide  use  of  evolutionary  development  approaches.  After  many  years  of  neglect, 
a  problem  as  complex  and  difficult  as  C2  modelling  requires  years  of  focused  research  and 
development.  There  are  no  simple  fixes  to  the  problem,  ft  is  evident  that  member  nations 
recognise  this  and  are  willing  to  approach  the  problem  in  an  incremental  manner,  applying 
evolutionary  approaches; 

•  There  is  progress  in  linking  and  federating  models.  Significant  progress  has  been  made  by  several 
nations  in  linking  performance  model-tools  with  combat  effectiveness  models,  either  directly  or 
through  off-line  approaches.  Additionally,  creating  federations  of  models  through  standard 
interface  protocols  has  significantly  improved  the  use,  and  reuse,  of  existing  models  and  has 
provided  a  promising  approach  for  future  modelling.  The  inherent  difficulties  in  federating 
models,  given  today’s  state  of  the  art,  must  be  considered  when  contemplating  this  approach; 

•  There  is  progress  in  modelling  “soft  factors”.  Several  nations  have  made  real  progress  in 
modelling  phenomena  that  have  non-physical,  or  soft  impact,  on  combat  operations  outcome. 
Among  these  factors  are  morale,  fatigue,  and  training  proficiency.  These  and  other  soft  factors 
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have  increased  importance  on  combat  operations  outcome  as  C2  modelling  improves  combat 
models; 

•  Standard  interface  protocols,  data  standards,  and  other  standards  either  now  exist,  or  are  under 
development.  These  standards  serve  to  make  this  difficult  task  easier.  Continued  development  of 
such  standards  is  envisioned  for  the  future  is  essential; 

•  There  is  widespread  use  of  Commercial-Off-The-Shelf  (COTS)  products.  These  products  are 
generally  available  to  all  member  nations.  This  use  of  COTS  has  served  to  help  further  standardise 
individual  modelling  approaches  and  will  continue  to  do  so  in  the  future;  and 

•  There  is  application  of  new  information  technologies.  New  technologies,  such  as  those  supporting 
animation,  have  been  applied  to  the  challenge  of  C2  modelling  simulation,  and  analysis. 
Additional  technological  advancements  will  no  doubt  continue  and  will  be  similarly  applied  to 
this  problem. 

8,5,2  Weaknesses  in  Current  C2  Tools/Modelling 

An  assessment  of  current  C2  modelling  approaches  employed  against  the  guidelines  above  show 
that  weaknesses  exist  in  current  approaches.  These  weaknesses,  rather  than  being  enumerated  here, 
are  expressed  as  challenges  below. 

8,6  RECOMMENDATIONS 

It  is  recommended  that  analysts  take  advantage  of  the  strengths  available  in  current  approaches  and  in  the 
new  methods  that  are  evolving.  They  should  also  be  aware  of  the  challenges  that  must  still  be  resolved  and 
should  attempt  to  play  their  part  in  helping  address  those  challenges  both  through  study  activities  and 
research. 

8,6,1  Challenges:  OOTW 

As  discussed  previously,  recent  world  events  and  current  projections  call  for  an  emphasis  on  OOTW. 
C2  in  these  environments  can  be  quite  different  and  may  require  fresh  C2  tool/modelling  approaches  to 
link  C2  to  outcome  in  these  environments.  The  following  challenges  with  tools-models  for  C2  Assessment 
exist: 


•  Orchestrating  a  set  of  applicable  tools.  Because  there  is  no  one  universally  accepted  tool  that  will 
satisfy  OOTW  C2  analysis  requirements,  a  set  of  tools  must  be  selected,  based  on  evaluation  of 
potential  tools  against  selection  criteria,  and  applied  to  the  analysis.  Orchestrating  a  set  of  tools 
that  complement  the  strengths  and  weaknesses  of  each  to  satisfy  the  analysis  requirements  is 
difficult.  Proper  consideration  of  assumptions  and  constraints  during  tool  selection,  and  careful 
scoping  the  analysis  issues,  will  help  to  simplify  the  orchestration  of  tools.  Additionally,  it  must 
be  remembered  that  not  only  the  tools,  but  trained,  skilled  users  of  the  tools  are  required; 

•  Breadth  of  tool  application.  Because  of  the  complexity  of  OOTW  analysis,  the  full  set  of  potential 
tools  should  be  considered  for  application  throughout  the  study  process,  from  problem 
formulation  through  sensitivity  analysis.  Additionally,  the  more  subjective  nature  of  OOTW 
dictates  that  tools,  heretofore  not  typically  applied  to  the  analysis  of  C2,  be  considered  for  use 
throughout  the  study  process; 

•  Relationship  of  tools  to  data  availability  and  MoM  generation.  Tools  must  be  selected  for  OOTW 
C2  analysis  that  have  necessary  data  available,  or  able  to  be  generated  or  obtained  from  the 
application  of  other  tools.  Reliance  on  more  subjective  data,  especially  for  higher  C2  echelons, 
may  be  necessary.  Additionally,  tools  must  be  selected  that  generate  the  MoMs  that  will  help 
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answer  the  study  issues  at  hand.  The  OOTW  C2  analyst  will  be  challenged  to  be  creative  in 
establishing  the  strong  relationship  between  data,  tools,  and  MoM  required  for  a  successful 
OOTW  C2  analysis.  It  must  be  remembered  that  analysts,  not  the  tools  themselves,  answer  the 
analysis  issues; 

•  Consideration  of  M-E-M  or  M-T-M.  Capitalisation  on  testing  or  training  events  for  C2  analysis 
purposes  can  be  highly  beneficial,  both  from  a  resource  and  analytic  point  of  view,  but  doing  so 
can  be  difficult.  Such  approaches  are  more  subjective,  take  more  human  involvement,  and  are 
inherently  more  complex  than  more  classical  analysis  tools/approaches.  The  advantages  of 
having  live  players/subjects  in  the  analysis,  however,  generally  outweigh  the  disadvantages. 
These  approaches  are  becoming  more  commonplace  and  more  accepted  as  a  legitimate  analysis 
approach,  especially  for  more  subjective  issues  such  as  those  associated  with  OOTW  C2; 

•  Sharing  of  tools  between  different  communities.  Given  the  nature  of  analysis  of  OOTW  C2  and 
the  emergence  of  M-E-M  and  M-T-M  approaches,  the  boundaries  between  the  testing,  training, 
analytic,  and  operational  communities  are  blurring  and  tools  once  considered  for  use  in  only  one 
of  these  communities  are  finding  application  in  another.  The  sharing  of  available  tools  among 
these  communities  is  considered  even  more  appropriate  now  with  the  rise  in  importance  of 
complex  OOTW  C2  analysis  requirements.  This  sharing,  however,  is  difficult  due  to  differences 
in  terminology,  as  well  as  cultural  differences  between  the  different  communities; 

•  Reuse  of  operational  schemas  and  data  models.  OOTW  typically  involves  many  more 
organisational  entities  than  combat  operations,  making  interoperability  of  organisations  and 
systems  critical  to  successful  operations.  C2  analysts,  therefore,  are  challenged  to  ensure  they  use, 
to  the  extent  possible,  existing  operational  schemas,  such  as  orders  and  reports,  and  data  models 
used  for  C3I  systems  integration,  whenever  possible  to  further  standardisation  and  interoperability 
goals; 

•  Analysis  of  architectures.  Technical,  operational,  and  system  architectures  are  sometimes 
developed  in  order  to  facilitate  integration  and  interoperability  of  C3I  systems.  For  NATO, 
required  architecture  frameworks  are  contained  in  the  NATO  C3  Interoperability  Management 
Plan  (NIMP),  Volume  II.  Considerable  challenges  exist  in  developing  and  applying  analytic  tools 
for  evaluation  architectures,  to  include  analysis  of  alternative  architectures  and  their  implications; 
and 

•  Management  of  customer  expectations  and  relations.  OOTW  C2  analysis  is  so  difficult  that  it  is 
necessary  for  the  analyst  to  ensure  the  customer  for  the  analysis  understands  the  inherent 
difficulty  and  to  attempt  to  manage  the  customer  expectations  for  the  analysis.  This  includes 
informing  the  customer  of  the  tools  to  be  employed  and  to  gain  an  understanding  and  trust  in 
those  tools  by  the  customer.  It  also  implies  a  special  relationship  between  analyst  and  customer 
for  OOTW  C2  analysis.  This  is  even  more  critical  when  the  customer  is  a  subject  of  the  analysis, 
such  as  when  a  commander  has  his/her  own  command  analysed. 

8,6,2  Challenges:  Modelling  C2 

The  following  challenges  with  modelling  C2  exist: 

•  Better  representation  of  cognitive  processes.  C2  can  be  incorporated  at  one  level  of  resolution  in 
combat  tools  through  representation  of  the  effects  of  particular  decisions.  At  another  level, 
representation  of  the  decision  process  itself  is  desirable.  It  would  enable  alternative 
decisionmaking  styles  and  the  effects  of  soft  factors  such  as  stress,  training  level,  fatigue, 
and  morale  to  be  more  easily  assessed.  These  factors  become  more  important  as  the  full  range  of 
10  representation  is  attempted; 
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•  Long  standing  challenges  associated  with  both  stochastic  and  deterministic  models. 
The  advantages  and  disadvantages  associated  with  stochastic  and  deterministic  modelling 
approaches  will  remain  as  C2  modelling  improves.  The  objective  is  to  select  the  best  modelling 
approach  for  the  issue  at  hand.  Recognising  the  inherent  advantages  and  disadvantages,  and  then 
capitalising  on  the  advantages  while  minimising  the  disadvantages,  are  the  challenges; 

•  Better  standard  definition  of  C2  terms.  This  challenge  has  plagued  the  C2  community  for  many 
years,  because  of  both  the  scope  and  complexity  of  the  subject.  This  is  particularly  true  across 
service  boundaries  and  across  the  international  community.  A  standard  set  of  definitions  would 
greatly  simplify  the  C2  modelling  challenge; 

•  The  definition  and  application  of  C2  scope.  This  challenge,  related  to  the  previous  challenge, 
is  especially  critical  to  modellers.  The  C2  of  a  fighter  aircraft  or  a  carrier  group  is  very  different 
from  the  C2  of  an  army  corps  or  an  army  squad.  On  the  other  hand,  there  are  C2  aspects  of  each 
of  these  combat  elements  that  are  similar.  Modelling  of  C2,  however,  can  be  vastly  different  in 
each  case.  The  scope  of  each  modelling  undertaking  must  be  properly  considered  and  discipline 
must  be  applied  throughout  model  development  to  focus  on  the  proper  scope.  Once  the  scope  is 
established,  a  mix  of  tools  may  be  required  to  address  the  full  scope  of  the  analysis; 

•  Multiple  application  of  C2  model-tools  to  analysis,  training,  and  operational  requirements. 
C2  phenomena  are  relatively  constant  whether  they  exist  within  the  analytic,  training, 
or  operational  environment,  and  they  should  be  consistently  modelled  in  each  environment. 
This  fact,  as  well  as  the  obvious  need  to  conserve  expensive  model-tool  development  resources 
wherever  possible,  leads  to  the  challenge  of  developing  C2  models,  or  at  least  component 
software  modules  that  can  be  used  to  support  analysis,  training,  and  operational  requirements; 

•  Level  of  resource  application  to  the  breadth  and  depth  of  C2  modelling.  Because  of  the 
large  scope  of  C2,  there  has  been  a  tendency  by  some  to  model  C2  at  great  breadth 
(multiple  applications),  at  the  expense  of  modelling  C2  phenomena  at  a  corresponding  depth.  In  a 
constrained  resource  world,  sufficient  resources  are  not  usually  available  for  both.  The  challenge 
is  to  either  apply  sufficient  resources  or  to  recognise  the  shortfall  and  to  level  the  available 
resources  across  the  breadth  and  depth  of  the  problem.  Models  and  other  tools  must,  therefore, 
be  tailored  to  the  extent  possible  to  fit  the  study  issues  being  addressed; 

•  Differences  in  the  level  of  modelling  of  friendly  and  adversary  forces.  Many  combat  models  do 
not  represent  adversary  forces  to  the  same  level  of  resolution  as  friendly  forces.  In  the  past, 
there  may  have  been  good  reasons  for  this.  Besides  the  obvious  resource  savings,  the  lack  of  C2 
representation  often  precluded  further  representation  of  adversary  forces.  Valid  representation  of 
C2,  to  include  full  play  of  10  such  as  deception  and  psychological  operations,  will  require  equal 
representation  across  both  adversary  and  friendly  forces,  as  well  as  any  other  supporting  or  neutral 
forces  in  the  simulation.  All  discussion  and  recommendations  in  the  COBP,  therefore,  are  equally 
applicable  to  modelling  of  adversary  forces  as  it  is  to  modelling  friendly  force  C2.  This  represents 
a  significant  challenge  to  many  modelling  efforts; 

•  Continuing  lack  of  “soft  factor”  representation  and  data.  As  discussed  previously,  a  robust  C2 
representation  in  combat  models  will  permit  soft  factors  to  be  better  represented.  The  bigger 
challenge,  perhaps,  may  not  be  the  modelling  methodology  itself,  but  the  acquisition  of  data  to 
support  it.  The  effects  of  such  things  as  stress,  training  proficiency,  morale,  fatigue,  and  shock, 
for  example,  necessitate  new  data  generation  approaches,  which  will  take  some  years  to 
implement.  The  tasks  of  VV&A  and  Verification,  Validation,  and  Certification  (VV&C)  are  most 
severe  in  their  soft  factor  arena.  The  certification  of  soft  factor  data,  as  well  as  most  all  C2-related 
data,  is  particularly  difficult  to  achieve.  Innovative  and  focused  C2  data  VV&C  programs  are 
required; 

•  VV&A  of  C2  model- tools  and  the  parameters  that  drive  them.  This  is  always  a  challenge  for 
model-tool  development  efforts,  but  is  particularly  challenging  for  C2  modelling,  due  to  the 
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variability  inherent  in  most  C2  processes,  especially  those  that  involve  the  human  aspects  of 
information  processing  and  decisionmaking;  and 

•  Sensitivity  Analysis.  The  challenges  associated  with  the  proper  conduct  of  sensitivity  analysis  of 
C2  is  as  great  as,  or  perhaps  greater  than,  that  associated  with  other  analyses.  This  is  because  of 
the  uncertainty  associated  with  C2  itself,  and  the  relatively  immature  modelling  of  C2  that  exists 
today.  Innovative,  yet  cost-effective  approaches  to  sensitivity  analysis  are  required. 


8.7  CHAPTER  8  ACRONYMS 

ACCESS 

Army  Command  and  Control  Evaluation  System 

AI 

Artificial  Intelligence 

C2 

Command  and  Control 

CPX 

Command  Post  Exercise 

FTX 

Field  Post  Exercise 

HEAT 

Headquarters  Effectiveness  Assessment  System 

HQ 

Headquarters 

MAPEX 

Map  Exercise 

M-E-M 

Model-Exercise-Model 

MoCE 

Measures  of  C2  Effectiveness 

MoFE 

Measures  of  Force  Effectiveness 

MoP 

Measures  of  Performance 

MoPE 

Measures  of  Policy  Effectiveness 

M-T-M 

Model-Test-Model 

ROE 

Rules  of  Engagement 

OOTW 

Operations  Other  Than  War 

STAFFEX 

Staff  Exercise 

TRADOC 

US  Army  Training  And  Doctrine  Command 

VV&A 

Verification,  Validation,  and  Accreditation 

vv&c 

Verification,  Validation,  &  Certification 
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A  non-linear, 
iterative  process 


—  Assess 
Risk 


Underlined  numbers  (3)  refer  to  chapters  of  the  COBP 
Shaded  box  indicates  the  current  chapter 


“Know  the  enemy  and  know  yourself;  in  a  hundred  battles  you  will  never  be  in  peril.  When  you  are 
ignorant  of  the  enemy  but  know  yourself  your  chances  of  winning  or  losing  are  equal.  If  ignorant  both  of 
your  enemy  and  of  yourself  you  are  certain  in  every  battles  to  be  in  peril.  ’’  -Sun  Tzu,  The  Art  of  War 


9.1  DEFINITIONS 

Data  are  factual  information  that  are  organised  for  analysis  and  in  a  form  suitable  for  machine  processing. 
Data  are  usually  thought  of  as  anchoring  an  epistemological  scale  with  understanding  or  wisdom 
anchoring  the  other  end.  Information  is  data  that  have  been  put  into  context. 

Metadata  are  “information  about  information.”  Metadata  can  describe  data  as  well  as  metadata,  therefore 
there  are  several  levels  of  metadata  possible.  An  example  for  metadata  on  a  higher  level  is  the  reliability  of 
the  source  that  data  have  been  derived  from.  In  general,  metadata  are  documentation  of  the  attributes  of 
data  directly  attached  to  the  data,  and  therefore  can  be  archived  along  with  the  data. 
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9.2  ROLE  OF  DATA 

The  role  and  importance  of  data  in  C2  assessment  is  underestimated  by  many  people,  often  including  the 
decisionmakers  and  the  assessment  team  itself  Figure  9.1  -  Data  Taxonomy  lays  out  a  number  of  the 
types  of  data  including  broad  categories  of  sources  that  will  be  of  interest  to  the  analyst.  The  ability  to 
determine  the  needed  data  and  the  ability  to  assemble  or  collect  this  data  determine  the  solution  strategy. 
The  capability  to  obtain  or  collect  the  appropriate  data: 

•  Will  often  reflect  the  difference  between  the  desired  or  “ideal”  measures  of  merit  (MoM)  and  the 
set  of  MoM  actually  available  and  used  in  the  assessment; 

•  Frequently  either  constrains  or  determines  the  scenario  or  set  of  scenarios  that  are  used; 

•  Is  a  key  or  major  factor  in  determining  the  set  of  tools  appropriate  for  the  assessment;  and 

•  Often  acts  as  a  schedule  and  cost  driver  for  the  analysis. 
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Figure  9.1:  Data  Taxonomy. 


9.3  REUSE  OF  DATA 

Because  of  the  centrality  of  data  to  the  assessment,  there  is  increasing  interest  in  data  reuse.  While  the 
amount  of  data  potentially  available  for  a  given  study  is  growing  exponentially,  the  real  opportunities  for 
the  reuse  of  data  have  proven  to  be  limited  because: 

•  The  rapid  change  of  technical  data  derailing  performance  of  systems  or  sub-systems  must  be 
identified  by  version  and  data; 

•  Assessment  teams  do  not  always  know  that  data  exist  and  have  no  easy  way  to  find  out  about  such 
“legacy”  data; 

•  The  data  that  are  available  are  seldom  in  an  easily  accessible  form; 
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•  The  conditions  under  which  the  data  were  collected  are  not  documented; 

•  The  definitions,  languages,  and  measurement  instruments  used  in  different  analyses  vary  widely; 
and 

•  Restrictions  arising  from  security  considerations. 

Despite  these  barriers,  an  effort  should  be  made  to  find  and  reuse  data.  Sources  that  should  be  considered 
include  “official  sources”  such  as  the  customer,  open  sources  such  as  those  available  on  the  Internet, 
and  prior  studies  on  similar  topics.  Because  the  data  needed  will  seldom  be  available  in  a  form  and  format 
ideal  for  the  assessment,  data  engineering  is  often  needed  to  gather,  organise,  and  transform  the  available 
data. 

Data  generated  by  a  given  assessment  phase  may,  in  itself,  be  valuable  as  an  input  later  in  the  research  or 
to  other  project  teams.  To  enable  appropriate  reusability  of  data,  every  modification,  constraint, 
assumption,  etc.  has  to  be  documented  adequately.  To  facilitate  the  reuse  of  data,  it  is  best  practice  to  use 
metadata  for  this  purpose. 

In  order  to  facilitate  the  reuse  of  data  across  community  boundaries,  alignment  of  the  processes  and 
methods  for  data  and  metadata  modelling  on  the  mid  term  -  resulting  in  shareable  data  and  metadata 
models  and  a  common  ontology  on  the  long  term  -  is  necessary. 

Furthermore,  the  issue  of  verification,  validation,  and  certification  (VV&C)  for  data  becomes  an  issue  of 
increasing  significance.  It  is  good  practise  to  use  certified  data  whenever  possible. 


9.4  DEFINITION  OF  DATA  DOMAINS 

Data  can  be  directly  connected  to  the  other  sections  of  the  Code  of  Best  Practice  (COBP)  by  respective 
data  domains.  These  data  domains  categorise  what  the  information  is  about  -  and  what  data  are  available  - 
as  well  as  the  assessment  needs  of  the  study  -  and  what  data  are  required.  Some  of  these  categories  are: 

•  Scenario  data:  the  set  of  data  describing  the  scenarios  and  vignettes; 

•  Human  organisational  issues  data:  the  set  of  data  describing  the  scenarios  and  vignettes; 

•  System  performance  data:  the  set  of  data  describing  system  performance  in  different  scenarios; 
and 

•  Tool  data:  the  set  of  additional  data  used  for  the  tools  that  are  not  covered  by  any  other  category. 
Hard-wired  assumptions  belong  here  as  well  as  study-specific  configuration  parameters  used  for 
technical  calibration  of  tools.  It  is  good  practice  to  be  aware  of  the  hidden  data  as  well  as  the  input 
parameters  for  each  tool  to  be  used  within  the  study,  especially  when  it  is  planned  to  build 
federations  of  tools. 

9.5  DATA  SOURCES 

Data  can  be  obtained  from  various  sources.  The  most  common  sources  include: 

•  Official  sources:  sources  such  as  military  databases,  governmental  data,  data  owned  by  the  United 
Nations,  etc.  The  customer,  or  other  stakeholders,  controls  the  access  to  this  data  or  is  at  least 
aware  of  the  existence  and  structure  of  the  data.  It  can  be  assumed  that  the  customer  will  support 
the  analyst  in  getting  access  to  the  data  or  a  sanitised  version,  in  case  the  original  data  are  not 
releasable  to  the  study; 
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•  Open  sources:  data  sources  that  are  neither  influenced  nor  controlled  by  the  customer. 
The  Internet,  commercial  organisations,  as  well  as  open  data  sources  of  non-participating 
organisations  are  examples; 

•  Legacy  study  results:  data  sources  derived  from  other  studies  conducted  by  the  operations  analysis 
and  operations  research  (OA/OR)  community,  including  political,  psychological,  and  sociological 
studies  and  Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance, 
and  Reconnaissance  (C4ISR)  tests  and  evaluations.  They  may  be  the  delivered  as  the  result  of  a 
former  study  as  well  as  having  been  an  intermediate  result;  and 

•  If  needed  data  are  not  available  from  empirical  sources,  subject  matter  experts  should  be  used  as 
sources  to  estimate  needed  values. 


9.6  DATA  CLASSES 

Data  classes  describe  the  technical  aspects  of  datum  format,  datum  types,  and  stage  of  processing.  Data 
classes  identified  in  the  context  of  this  section  are  the  following: 

•  Raw  data  are  unprocessed.  Raw  data  may  come  from  observations  of  reality  or  artificial  reality, 
the  product  of  an  instrumented  reality,  experimental  situations,  or  selected  artificial  realities; 

•  Processed  data  or  information  are  the  result  of  transforming  one  or  more  raw  data  elements  into 
another  variable.  For  example,  one  or  more  radar  returns  are  transformed  into  a  track  (friendly  or 
not); 

•  Aggregate  data  or  information  are  properties  of  a  collection  of  elements.  For  example, 
the  movement  of  individuals  on  a  battlefield  versus  the  movement  of  a  squad  or  a  platoon; 

•  Statistical  values'  are  assessed  on  a  sample  of  a  population  and  characterised  by  this  population. 
These  include  mean,  modes,  medians,  standard  deviations,  and  kurtosis.  Statistics  are  often  used 
as  parameters  in  assessments; 

•  Derived  data  or  information  are  outputs  from  a  formula  or  simulation  model  that  implicitly 
incorporates  a  set  of  assumptions.  For  example,  loss-exchange  ratios  given  sensor  and  weapons 
assumptions;  and 

•  Intermediate  data  and  information  are  the  products  of  one  phase  or  component  of  the  assessment 
that  provides  input  to  another  phase  or  component. 

Some  C2  assessments  use  assumptions  or  preset  parameters  in  place  of  data  or  statistics. 


9.7  USE  OF  METADATA 

Data  that  are  collected  without  adequate  documentation  are  frequently  viewed  as  suspect.  To  avoid  having 
good  data  discarded  due  to  a  lack  of  documentation,  acceptable  community  standards  for  documentation 
should  be  employed.  The  data  must  be  clearly  described  in  a  manner  that  is  understandable  to  a  subject 
matter  expert,  not  directly  involved  with  conducting  the  study.  The  data  description  must  be  robust  enough 
to  inspire  user  confidence  in  the  data.  It  is  good  practice  to  record  these  considerations  in  the  metadata 
associated  with  the  data  to  facilitate  the  data  use  and  reuse. 

Data  may  be  in  any  of  the  relevant  levels  of  measurement  -  ratio,  interval,  ordinal,  or  nominal.  In  C2 
assessments,  some  significant  factors  may  be  nominal.  It  is  good  practise  to  record  all  of  the  assumptions, 
constraints,  and  limitations  in  the  metadata. 


'  Statistics  is  a  branch  of  mathematics  dealing  with  the  collection,  analysis,  interpretation,  and  presentation  of  masses  of 
numerical  data;  alternatively,  statistics  is  a  collection  of  quantitative  data  (Webster  Online  Dictionary). 
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In  general,  every  time  data  are  modified  or  proeessed  to  ereate  new  data,  appropriate 
metadata  doeumentation  should  be  provided  to  ensure  traeeability  of  results,  validation,  verifieation, 
and  eertifieation  of  respeetive  data,  and  reuse  of  data  in  later  phases  or  within  other  studies.  This  also 
allows  dealing  with  the  ehallenge  of  multiple  instanees  of  data  in  different  studies. 


9.8  DATA  AND  PROBLEM  FORMULATION 

The  initial  data  available  will  often  be  vague,  uneertain,  ineomplete,  and  eontradietory.  Analysts  usually 
prefer  data  to  be  sharp,  eertain,  complete,  and  consistent.  It  is  necessary  to  be  explicit  about  the 
assumptions  inherent  in  this  transformation. 

If  the  assessment  team  finds  it  necessary  to  transform  “soft”  data  to  “hard”  data  in  order  to  use  tools  that 
require  “hard”  data,  it  is  best  practice  to  record  the  source,  perceptions  etc.  in  the  metadata  associated  with 
the  data. 

9,8,1  Obtaining  Data 

It  may  be  safely  assumed  that  not  all  of  the  required  data  will  be  available  pre-packaged  for  the  study 
question.  Some  relevant  data  will  likely  be  submerged  among  a  pool  of  irrelevant  information. 

Not  all  data  will  be  under  military/government  control.  Data  belong  to  the  stakeholders  who  are  not 
necessarily  connected  to  (or  even  friendly  toward)  the  customer  of  the  study.  A  lot  of  information  is 
available  from  open  sources.  The  challenge  is  to  find,  organise,  verify,  process,  and  convert  it  into  the  data 
needed.  The  team  should  be  aggressive  and  persistent  in  the  pursuit  of  required  data. 

The  data  needed  will  seldom  be  available  in  its  raw  form.  Often,  data  have  been  transformed  and 
aggregated.  When  tools  have  to  be  applied  to  derive  data,  the  derived  data  should  be  tagged  with 
explanatory  metadata  to  record  that  information  in  a  form  that  will  facilitate  its  reuse. 

If  the  data  are  not  available  and  can  neither  be  aggregated  nor  derived  from  the  available  sources,  it  is 
good  practice  to  use  the  knowledge  of  subject  matter  experts  to  generate  the  necessary  data.  Increasingly, 
when  C2  assessment  teams  are  tasked  with  exploring  new  concepts  of  operation,  empirical  data  can  not  be 
expected  to  exist.  It  is  further  best  practice  to  document  this  in  respective  metadata  and  replace  such 
assumption  based  data  as  early  as  possible  with  empirical  data,  e.g.,  as  soon  as  another  study  delivers  the 
needed  inputs.  It  is  recommended  to  check  the  respective  study  results  if  subjectively  generated  data  are 
replaced,  particularly  when  the  study  result  has  been  shown  to  be  sensitive  to  this  data  element. 

The  team  needs  to  know: 

•  What  data  are  needed  /  structure  of  data: 

•  Preferred  data  (independent  variables  within  the  MoM); 

•  Necessary  data  (to  be  able  to  drive  the  tools); 

•  Available  data  (derivation,  extraction,  collection,  etc.); 

•  Who  owns  this  data; 

•  Security  issues,  possibility  of  declassifying  or  otherwise  obtaining  release  may  be  another  issue; 

•  Costs  to: 

•  Buy  data; 

•  Collect  data  (people,  time,  resources);  and 

•  Generate  data. 
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It  is  essential  to  make  the  value  of  data  elear  to  all  levels  of  decisionmakers  and  operational  planers  to 
ensure  that  data  collection  issues  are  included  in  all  phases  of  the  study.  The  data  collection  and 
engineering  plan,  as  introduced  in  section  4-E-4,  as  part  of  the  solution  strategies,  has  to  take  this  into 
account. 

The  process  of  data  acquisition  is  important,  but  data  acquisition  should  not  be  emphasised  over  data 
interpretation. 

There  is  an  increasing  urgent  need  for  data  describing  operations  over  the  complete  mission  spectrum.  It  is 
good  practise  to  collect  and  exploit  operational  data  whenever  possible.  It  is  therefore  recommended  to 
synchronise  respective  data  collection  and  engineering  plans,  ensuring  from  the  start  that  the  desired  data 
will  be  collected  and  archived  appropriately. 

9,8.2  The  Use  of  Data  within  the  Study 

The  archiving  of  data  in  retrievable  form  is  essential.  This  is  necessary  both  to  support  the  ongoing  study 
and  also  to  be  of  value  for  future  study  efforts.  The  team  should  establish  and  adopt  process  models  that 
ensure  the  build-up  of  archives  within  a  respective  infrastructure.  Metadata  have  a  critical  role  to  play  in 
data  archiving  and  retrieval. 

The  discussion  of  data,  and  the  development  of  a  community  database,  must  be  driven  by  agreed  upon 
definitions  of  data  that  are  sufficient  to  support  the  community.  This  should  be  a  high  priority  item. 

Data  can  be  described  as  the  “glue”  that  holds  together  the  different  phases  of  a  study.  As  the  results  from 
one  phase  are  transported  to  the  next  it  is  unlikely  that  both  will  use  identical  processes  and  procedures, 
therefore  harmonised  data  formats  will  allow  for  smooth  transition  and  continuity  of  effort. 

The  analyst  will  be  faced  with  data  in  various  forms  and  formats.  In  order  to  consider  all  available  data, 
find  it  if  necessary,  and  format  it  in  the  required  manner  (e.g.  as  an  input  parameter  for  the  MoM  or  a 
model)  data  must  be  stored  appropriately.  One  standard  for  storing  data  is  the  Information  Resource 
Dictionary  Systems  (IRDS)  [ISO  1990]. 

IRDS  is  a  layered  database  that  not  only  comprises  the  data,  but  also  the  metadata  describing  the  meaning 
of  the  data,  the  format,  the  constraints,  the  intended  use,  the  source,  degree  of  certainty,  vagueness  and 
reliability,  etc.  It  comprises  data  as  well  as  metadata. 

A  common  understanding  of  the  problem  between  the  customer  and  the  study  group  as  well  as  among  the 
interdisciplinary  study  team  is  essential  for  the  success  of  the  study.  Therefore,  a  study  glossary  -  based  on 
a  general  and  evolving  OR  glossary  (e.g.  the  NATO  JOINT  Pub  1-02  [US  DoD  1999])  -  is  needed. 
The  data  definitions  stored  as  metadata  have  to  be  aligned  with  the  definitions  found  in  the  study  glossary. 
This  provides  a  basis  for  standardised  documentation  of  study  results  in  a  highly  reusable  form  that  can  be 
manipulated  easily  and  reliably,  perhaps  by  automated  systems. 

Using  the  right  toolkit  for  the  management  of  data  within  the  IRDS,  such  as  the  Shared  Data  Environment 
(SHADE),  can  create  the  initial  state  of  every  future  data  driven  application  (DISA,  1996;  Krusche  and 
Tolk,  2000).  The  same  techniques  and  tools  can  and  should  be  used  for  information  systems  delivering  the 
needed  functionality  to  the  warfighter  and  decisionmaker  (Tolk,  2000). 

9.9  CONCLUSIONS  AND  RECOMMENDATIONS 

Data  transcend  all  phases  of  an  assessment  and  may  be  seen  as  the  “glue”  that  holds  the  phases  of  an 
assessment  together.  Given  its  importance,  resources  must  be  committed  to  ensure  effective  data 
acquisition,  management,  and  availability  for  reuse. 
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There  is  an  urgent  need  to  agree  on  standards  for  data,  metadata,  and  data  management,  including  the 
conditions  under  which  the  data  are  collected,  data  element  definitions,  metrics,  etc.  It  is  good  practise  to 
use  established  standards,  where  appropriate,  such  as  the  Source  for  Environmental  Representation  and 
Interchange  (SEDRIS^’^),  or  to  use  de  facto  standards  like  Digital  Terrain  Elevation  Data  (DEED). 

As  the  data  being  used  today  by  the  analysts  will  be  the  data  needed  tomorrow  by  systems  engineers, 
decisionmakers,  and  commanders  for  their  operations,  alignment  of  the  standardisation  processes  and  the 
respective  toolsets  as  early  as  possible  with  the  command  and  control  systems  community  is  good 
practice.  A  significant  first  step  in  such  an  alignment  would  be  using  the  same  IRDS  [NATO  1999]. 
A  common  C4I  and  Modelling  and  Simulation  (M&S)  community  is  needed,  to  make  visions  like 
integrated  alternative  course  of  action  analyses  become  a  reality. 


9.10  CHAPTER  9  ACRONYMS 

C2  Command  and  Control 

COBP  Code  of  Best  Practice 

C4ISR  Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance, 

and  Reconnaissance 

IRDS  Information  Resource  Dictionary  Systems 

MoM  Measures  of  Merit 

M&S  Modelling  and  Simulation 

OA/OR  Operations  Analysis  and  Operations  Research 

SHADE  Shared  Data  Environment 

9.11  CHAPTER  9  REFERENCES 

Defense  Information  Infrastructure  (Dll)  Shared  Data  Environment  (SHADE),  Capstone  Document. 
(1996).  Arlington,  VA:  U.S.  DoD,  Defense  Information  Systems  Agency  (DISA).  Available  from 
http://www.disa.mil/coe/srs/srs_base/srs_csa_PDF/zsrscsa.pdf 

The  Federation  of  American  Scientists  (FAS)  Web  Page  on  Digital  Terrain  Elevation  Data: 
http://www.fas.org/irp/program/core/dted.htm. 

IS10027:  1990  an  Information  Resource  Dictionary  System  (IRDS)  Framework.  (1990). 

Krusche,  S.,  &  Tolk,  A.  (2000,  October).  Information  Processing  as  a  Key  Factor  for  Modern  Federations 
of  Combat  Information  Systems.  Paper  presented  at  the  North  Atlantic  Treaty  Organisation  (NATO) 
Research  &  Technology  Organisation  (RTO)  Information  Systems  Technology  Panel  (1ST)  Symposium: 
New  Information  Processing  Techniques  for  Military  Systems,  Istanbul,  Turkey.  Available  in  PDF  format 
from  http://home.t-online.de/home/Andreas.Tolk/literat.htm. 

Eambert,  N.J,  (1997).  ''Operation  Firm  Endeavour:  Campaign  Data,  Its  Collection  and  Use.”  Presented  at 
the  Directorate  of  Eand  Warfare  Historical  Co-ordination  Group  Conference,  Trenchard  Fines,  DGD&D, 
Upavon,  UK,  28-29  April  1997.  Also  available  from  the  British  Eibrary  Document  Supply  Centre,  Boston 
Spa,  Wetherby,  West  Yorkshire,  ES23  7BQ,  United  Kingdom,  Reference  DSC  Shelfmark:  99/20363. 

Mathieson,  G.E.,  Tolk,  A.,  Sundfoer,  H.O.,  &  Sinclair,  M.  (2001).  Data  the  Glue  to  Assessment. 
Washington,  DC:  Cooperative  Research  Program  (CCRP). 


RTO-TR-081 


9-7 


DATA 


ORGAmZATION 


North  Atlantic  Treaty  Organisation  (NATO)  Consultation,  Command  and  Control  Board  (NC3B), 
Information  Systems  Sub-Committee  (ISSC),  NATO  Data  Administration  Group  (NDAG).  (1999). 
Feasibility  study  for  the  NATO  C3  Information  Resource  Dictionary  System,  (NC3  Information 
Resource  Dictionary  System  (IRDS),  AC/322SC/5-WG/3WP4).  Brussels,  Belgium.  Available  from 
http//www.nato.int/docu/home.htm. 

The  Report  of  the  Data  Working  Group.  (1998).  In  Simulation  Technology-1997  (SIMTECH  97,  Vol.  III). 
Alexandria,  VA:  Military  Operations  Research  Society. 

Sinclair,  M.,  Tolk,  A.  (2002,  April).  Building  up  a  Common  Data  Infrastructure.  NATO  Symposium 
SAS039  on  the  “Analysis  of  the  Military  Effectiveness  of  Future  C2  Concepts  and  Systems”  NATO  C3 
Agency,  The  Hague,  The  Netherlands. 

The  Source  for  Environmental  Representation  and  Interchange  (SEDRIS^'^)  Home  Page: 
http  ://www.  sedr  is .  com. 

Tolk,  A.  (2000,  October).  Functional  Categories  of  Support  to  Operations  in  Military  Information 
Systems.  Proceedings  of  the  North  Atlantic  Treaty  Organisation  (NATO)  Regional  Conference  on 
Military  Communication  and  Information  Systems,  Zegrze,  Poland.  Available  in  PDF  format  from 
http://home.t-online.de/home/Andreas.Tolk/literat.htm. 

Tolk,  A.  (2001,  March).  Bridging  the  Data  Gap  —  Recommendations  for  Short,  Medium,  and  Long 
Term  Solutions,  (OlS-SIW-01 1).  Proceedings  of  the  Simulation  Interoperability  Workshop  Spring 
2001,  Orlando,  Florida.  Available  in  PDF  format  from  http://home.t-online.de/home/Andreas.Tolk/ 
011_0301.pdf 

Tolk,  A.,  &  Kunde,  D.  (2000,  June).  Decision  Support  Systems  -  Technical  Prerequisites  and  Military 
Requirements.  Proceedings  of  the  2000  Command  and  Control  Research  and  Technology  Symposium 
(CCRTS),  Monterey,  CA.  Available  in  PDF  format  from  http://www.dodccrp.org/6thICCRTS/Cd/Tracks/ 
Papers/Track6/03  8_tr6.pdf 

US  Department  of  Defense  (DOD),  Joint  Chiefs  of  Staff  (1999).  Dictionary  of  Military  and  Associated 
Terms.  Washington,  DC:  Greenhill  Books.  Available  from  http://www.dtic.miFdoctrine/jel/doddict/ 
index.html. 


9-8 


RTO-TR-081 


Chapter  10  -  RISK  AND  UNCERTAINTY 


ORGAmZATIOM 


A  non-linear, 
iterative  process 


Underlined  numbers  (3)  refer  to  chapters  of  the  COBP 
Shaded  box  indicates  the  current  chapter 


“We  may  at  once  admit  that  any  inference  from  the  particular  to  the  general  must  be  attended  with  some 
degree  of  uncertainty,  but  this  is  not  the  same  as  to  admit  that  such  inference  cannot  be  absolutely 
rigorous,  for  the  nature  and  degree  of  the  uncertainty  may  itself  be  capable  of  rigorous  expression.  ” 
-R.A.  Fisher,  The  Desisn  of  Experiments 

RISK  AND  UNCERTAINTY 

There  are  risks  associated  with  the  decisionmaker’s  situation  that  are  an  inherent  part  of  the  analysis. 
There  are  also  risks  related  to  the  conduct  of  the  analysis  itself  This  chapter  deals  with  both  of  these  sets 
of  risks,  focusing  upon  reducing  uncertainty  and  other  contributors  to  risk  as  well  as  the  mitigation  of  their 
effects.  Failure  to  deal  effectively  with  risks  will  jeopardise  the  accomplishment  of  the  goals  of  the  study, 
namely  to  provide  high  quality  decision  support.  It  should  be  noted  that,  by  the  end  of  the  assessment, 
the  team  will  generally  be  aware  that  the  initial  study  plan  will  have  been  changed,  whereas  flaws  in  the 
conceptual  model  or  in  the  assessment  design  may  remain  hidden  for  some  time  from  both  the  analyst  and 
the  decisionmaker. 

While  the  adoption  of  the  guidance  contained  in  this  Code  of  Best  Practice  (COBP)  will  help  minimise  the 
risks,  it  will  not  eliminate  them.  This  chapter  discusses  a  number  of  issues  related  to  the  risks  and 
uncertainties  that  the  assessment  team  needs  to  explicitly  address. 
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10.1  RISK 

Risk  is  commonly  defined  as  the  possibility  of  suffering  harm  or  loss;  in  other  words,  an  exposure  to  harm 
or  loss'.  This  includes  an  opportunity  loss.  The  difference  between  risk  and  loss  is  that  risk  inherently 
involves  before  the  fact  probability,  while  loss  is  an  after  the  fact  certainty.  Insurance  is  about  estimating 
and  covering  an  exposure  of  value  to  uncertainty.  Risk  often  has  a  negative  connotation,  yet  “taking  risks” 
can  also  be  a  positive  act  when  a  proper  balance  or  trade-off  is  made  between  good  and  bad  outcomes  and 
the  respective  probabilities  associated  with  the  outcomes. 

Perceptions  of  risk  can  substantially  differ  from  more  objective  assessments  of  probability  and  impact. 
This  is  particularly  important  in  command  and  control  (C2)  problems,  which  have  a  high  sociological 
content. 

10.2  UNCERTAINTY 

Uncertainty  can  be  generally  defined  as  an  inability  to  determine  a  variable  value  or  system  state 
(or  nature)  or  to  predict  its  future  evolution.  Uncertainty  is  inherent  in  risk.  Even  if  a  person  is  certain 
about  the  possible  outcomes  and  the  probabilities  associated  with  these  outcomes,  there  is  uncertainty 
about  which  outcome  will  in  fact  occur.  When  this  situation  applies  we  can  say  that  we  have  a  known  risk. 
For  example,  the  possible  outcomes  and  the  probabilities  associated  with  the  toss  of  a  coin  (and  many 
other  popular  forms  of  gambling)  are  known.  What  is  uncertain  is  the  outcome.  Hence  there  is  a  risk, 
in  this  case  a  known  risk,  associated  with  obtaining  a  particular  outcome. 

There  are  other  types  of  uncertainty,  including  uncertainty  about  the  possible  outcomes  (or  their  values) 
and  uncertainty  about  the  probabilities  associated  with  these  outcomes.  Uncertainties  arise  from  other 
uncertainties,  namely  uncertainties  about  the  potential  actions  of  others  or  what  is  referred  to  as 
“states  of  nature”  and  their  associated  probabilities.  In  C2-related  assessments,  in  particular,  perceptions  of 
uncertainty  may  substantially  differ  from  a  more  objective  assessment  of  lack  of  knowledge. 

There  are  also  a  number  of  uncertainties  that  are  associated  with  the  assessment  itself.  These  are  discussed 
later  in  this  chapter. 


10.3  DEALING  WITH  RISK 

There  are  three  basic  ways  to  deal  with  risk.  The  first  is  to  reduce  the  uncertainty  that  underlies  the  risk. 
The  second  is  to  mitigate  risk  by  developing  and  selecting  risk-averse  strategies.  The  third  is  to  effectively 
communicate  the  nature  of  the  risk  involved. 


10.4  REDUCING  UNCERTAINTIES 

In  effect,  all  methods  for  reducing  the  uncertainties  that  underlie  risk  involve  the  collection  or  analysis  of 
information.  This  is,  of  course,  a  major  focus  of  an  assessment.  Clearly  the  assessment  team  needs  to 
identify  the  most  important  uncertainties  -  a  combination  of  the  degree  of  uncertainty  and  the 
consequences  of  that  uncertainty.  However,  there  will  always  be  some  significant  residual  uncertainties. 
The  aim  of  the  assessment  team,  at  this  point,  is  to  reduce  the  risk  associated  with  the  study  by  learning 
more  about  the  robustness  (or  lack  thereof)  of  the  study  findings  and  conclusion.  Sensitivity  analysis  is 
one  method  to  accomplish  this  purpose. 


'  This  interpretation  of  risk  is  used  throughout  this  section.  However,  other  common  definitions  have  been  included  at  Annex  E 
as  an  aid  to  common  understanding. 
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10,4,1  Sensitivity  Analysis 

The  goals  of  sensitivity  analysis  are  fourfold: 

•  To  establish  the  regions  for  which  established  results  are  valid; 

•  To  isolate  those  factors  that  contribute  most  to  the  uncertainties  that  exist  and  identify  the  risks 

associated  with  these  uncertainties  as  they  relate  to  study  findings  and  conclusions; 

•  To  make  study  results  more  robust  for  decision  support  by  allowing  the  decisionmaker  to  see  the 
immediate  consequences  as  external  factors  are  changing;  and 

•  To  give  the  decisionmaker  a  richer  understanding  of  the  decision  problem,  highlighting  the 
consequences  of  limited  changes  relative  to  a  solution  proposed  by  the  study. 

The  team  should  take  a  three-step  approach: 

•  Identify  the  variables  that  are  associated  with  the  greatest  combination  of  risk  and  uncertainty; 

•  Perform  sensitivity  analyses  that  vary  across  the  more  likely  region  of  key  parameters  (often  the 

neighbourhood  of  the  initial  estimate)  to  see  how  the  result  is  influenced.  A  sensitivity  analysis 

investigates  the  main  region  of  the  output  space;  and 

•  Investigate  the  extremes  of  the  same  output  space,  focusing  most  strongly  on  the  negative  regions. 

The  assessment  team  also  partially  takes  an  inverted  approach,  looking  for  possible  failures,  then  seeking 
possible  sources  of  each  kind  of  failure. 

Together,  these  three  steps  should  provide  the  team  with  a  better  understanding  of  residual  uncertainties 
and  associated  risks.  The  need  for  and  results  of  sensitivity  analyses  must  be  stressed  in  discussions  with 
the  decisionmakers.  This  will  help  avoid  the  under-resourcing  of  this  critical  activity. 


10,5  UNCERTAINTY  AND  RISK  IN  C2  ASSESSMENTS 

Uncertainty  and  the  risks  associated  with  it  can  not  be  totally  eliminated  in  any  real  world  C2  assessment 
problem.  Moreover,  in  most  real-life  problems,  one  cannot  even  identify  all  of  the  unknowns.  This  has 
given  rise  to  the  term  the  “unknown  unknowns.”  Strategies  to  minimise  risk  have  evolved  to  handle  these 
facts  of  life.  Therefore,  even  the  best  possible  assessment  approach  will  result  in  residual  uncertainty  and 
risks. 

In  general,  assessments  should  be  judged  by  their  ability  to  reduce  uncertainty  so  that  the 
decisionmaker  is  in  a  better  position  (less  risk)  after  the  assessment  than  before  the  assessment. 

Looking  at  the  absolute  uncertainty  that  remains  is  not  as  useful  a  measure  since  it  may  say  more  about  the 
nature  of  the  problem  than  the  success  of  the  assessment. 

10,5,1  Dealing  with  Uncertainty 

It  is  important  to  treat  uncertainty  consistently  and  explicitly.  This  allows  information  from  two  given 
sources  or  results  to  be  fused,  (e.g.  by  taking  the  most  precise  assessment  of  each  factor  from  the  two 
results).  Thus  the  resulting  knowledge  will  be  better  than  either  of  the  two  separate  results.  On  the  other 
hand,  if  uncertainty  is  not  treated  explicitly  or  consistently,  the  best  one  can  do  is  to  pick  the  single  result 
that  seems  best.  This  makes  it  more  difficult  for  a  study  to  add  value  to  a  decisionmaker. 

C2  issues  are  complex,  and  it  is  an  understatement  to  say  they  are  incompletely  understood. 
C2  requirements  and  solutions  tend  to  depend  heavily  on  the  nature  of  the  operating  environment. 
Thus,  a  C2-related  study  will  rely  on  factors  that  are  imprecisely  determined  and  change  frequently  over 
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time.  Almost  regardless  of  the  assessment  effort,  parts  of  the  problem-space  will  have  been  investigated 
less  thoroughly  than  would  be  ideal.  Sensitivity  analyses  are  required  to  give  high  quality  study  results. 

C2  assessment  problems  (particularly  in  operations  other  than  war  (OOTW)  contexts)  generally  have 
many  interacting  factors.  It  is  unwise  to  rely  upon  single  factor  sensitivity  analysis  (i.e.  testing  sensitivity 
to  one  factor  at  a  time).  Multi-factorial  experimental  design  methods  are  good  practice  in  such 
circumstances  (Keppel,  1973)^. 

OOTW  studies  typically  have  less  well-formed  quantitative  factors  and  more  qualitative  factors. 
The  “softer”  nature  of  these  factors  makes  assessment  more  difficult.  The  types  of  factor  seen  in  OOTW 
problems  include: 

•  Social  and  political  activity  impacting  the  tactical  level; 

•  A  strong  influence  of  negotiation  and  persuasion  as  opposed  to  coercion; 

•  Non-optimal  performance  of  military  capabilities  from  a  technical  perspective  due  to  their  poor  fit 
to  the  problem; 

•  Severe  rules  of  engagement  (ROE)  constraints;  and 

•  Unclear  or  evolving  goals  and  objectives. 

The  team  must  be  aware  of  the  assumptions  and  limitations  included  in  models,  scenarios,  and  data 
structures,  which  should  be  captured  in  the  metadata.  In  particular,  it  should  be  noted  that  humans 
involved  in  C2  experiments  (as  analysts  or  subjects)  always  bring  assumptions  with  them.  These  need  to 
be  identified  and  collected  to  form  an  audit  trail.  This  is  one  place  where  the  project  leader’s  log  of 
assumptions  and  decisions  made  during  the  assessment  pays  off.  Uncertainty  over  the  validity  of  these 
assumptions  and  limitations  provide  a  source  of  uncertainty  in  study  results  to  which  study  conclusions 
must  be  made  robust. 

In  C2  assessments,  in  particular,  all  aspects  of  a  study  and  study-problem  may  be  connected  to  uncertainty. 
Thus,  different  sorts  of  uncertainty  should  be  addressed  explicitly  at  appropriate  stages  in  the  study. 
Examples  of  these  are: 

•  Parameter  value  uncertainty  -  many  of  the  parameters  and  factors  in  C2  assessment  are  difficult  to 
evaluate  confidently; 

•  Model-based  uncertainty  -  i.e.,  over  whether  underlying  models  are  valid  and  representative; 

•  Uncertainty  of  focus  -  i.e.,  over  whether  the  assessment  covers  all  of  the  important  factors  and/or 
issues  (this  includes  uncertainty  of  scenarios);  and 

•  Complexity  of  uncertain  factors  (i.e.  their  dimensionality)  -  when  a  sufficiently  complex  factor 
(e.g.  scenarios  or  future  technology)  is  uncertain,  the  team  can  not  expect  to  overview  the  set  of 
all  possible  true  states. 

Uncertainty  in  complex  factors,  such  as  scenarios,  should  be  addressed  thoroughly.  Even  though  it  is 
philosophically  impossible  to  know  everything  about  a  problem,  an  adequately  complete  knowledge  can 
be  better  assured  by  explicit  use  of  checklists  that  highlight  the  breadth  of  factors  typically  involved  in  C2 
assessments,  such  as: 

•  Technology  -  disruptive  uncertainty  and  disruptive  innovation; 

•  Organisational  use  of  technology; 

•  Scenarios,  tasks,  and  nature  of  operations; 


2 
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•  Data; 

•  Context  or  environment  of  the  assessment;  and 

•  Co-evolution  of  factors  (e.g.  summarised  in  doctrine,  organisation,  training,  material,  leadership, 
personnel,  and  facilities  [DOTMLPF]). 

Nonetheless,  the  team  should  not  rely  completely  on  checklists,  but  rather  complement  them  with  critical 
thinking  in  the  specific  study  context. 

Within  electronic  systems,  organisations,  and  battle  concepts,  there  is  a  lot  of  opportunity  for  disruptive 
technology,  producing  substantial  uncertainty.  In  human-in-the-loop  experiments  there  is  a  greater  danger 
of  bias  in  subjective  judgements^ 

In  considering  sensitivity  analysis,  it  is  important  not  to  associate  it  only  with  statistical  variance  in 
parameter  values.  Qualitative  consideration,  such  as  variation  of  model,  perspective,  or  assumptions 
(i.e.  categorical  variations)  should  also  be  used  to  test  for  and  assess  sensitivity.  Variations  in  ranking 
(ordinal  variation)  can  also  be  a  powerful  tool  for  sensitivity  analysis.  Other  analytic  tools  and  constructs 
relevant  to  sensitivity  analysis  are: 

•  Non-parametric  statistics  (Siegal  and  Castellan,  1988); 

•  Belief-functions  (as  an  alternative  to  probability); 

•  Judgmental  uncertainties  (Wilson  and  Corlett,  1990); 

•  Fuzzy  numbers,  theories  of  semiorders,  scoring  criteria  (Siegal  and  Castellan,  1998); 

•  Multi-dimensional  scaling;  and 

•  Mathematics  applied  to  non-ordinal  scales. 

Semiorders  constitute  the  intermediate  level  between  ordinal  information  and  value.  They  apply  to  many 
fields  but  in  this  context  especially  to  scored  data  or  preference  data  -  in  that  account  is  taken  of  the 
intervals  of  imprecision  around  the  measuring  systems  used.  Analysts  should  be  aware  of  the 
thresholds  associated  with  the  collected  data  above  which  differences  can  be  legitimately  distinguished  to 
produce  values.  This  idea  is  very  useful  when  the  measurements  cannot  be  repeated  (as  in  statistical 
theory)  (Prilot  and  Vincke,  1997). 

Keeping  C2  assessment  rigorous  and  robust  in  the  face  of  the  many  uncertainties  and  complexities  of  the 
subject  matter,  as  well  as  the  need  to  use  a  rich  combination  methods,  can  be  difficult.  Again,  it  is  good 
practice  to  use  checklists  and  structured  appraisal,  in  this  case  to  maintain  an  objective  view  of  study 
rigour.  The  choice  of  checklist  or  appraisal  structure  can  depend  upon  personal  preferences,  but  Annex  F 
lists  a  number  of  structures,  which  have  proved  useful  in  the  experience  of  the  nations  contributing  to  this 
COBP.  These  include: 

•  Repeatability,  independence,  grounding  in  reality,  objectivity  of  process,  uncertainty  and 
robustness  (RIGOUR);  and 

•  Strengths,  weaknesses,  opportunities,  and  threats  (SWOT). 

Where  an  assessment  uses  experimentation  or  observation  of  exercises  it  is  important  to  identify 
independent  and  dependent  variables  and,  for  the  former,  which  are  controllable,  which  are  measurable, 
etc.  Figure  10.1,  for  example,  illustrates  the  variety  of  variables  that  need  to  be  considered  when  studying 
decisionmaking. 


^  These  types  of  experiment  are  typically  unrepeatable  due  to  resource  constraints. 
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INPUTS 


Persongl 

-  Culture 

-  Style 

-  etc. 


Institutional 

-  Culture 

-  Doctrine 

-  etc. 


.CONSTRAINTS 


OUTPUTS 


“Sensor”  Information 
(with  variability) 


Decision  Making 


RESOURCES 


Action  Instructions 
(with  variability) 


Personal 

-  memory 

-  experience/knowledge 

-  skill/expertise 

-  etc. 


Insititutional 

-  memory 

-  information  technology 

-  administrative  support 

-  etc. 


Figure  10.1:  Illustration  of  the  Variety  of  Variables  Relevant  to  Decisionmaking. 

A  thorough  understanding  of  the  variables  of  a  study  is  essential  for  effective  treatment  for  uncertainty. 

10,5,2  Risk-Based  Analysis 

In  cost-benefit  or  cost-effectiveness  analyses  there  has  historically  been  a  tendency  to  focus  on  the  single 
most  likely  value  (its  expected  value)  for  each  input  factor  (including  scenario  and  course  of  action)  and  to 
“solve”  the  problem  based  upon  these  point  estimates.  This  can  lead  to  fragile  solutions,  which  do  not 
provide  decisionmakers  with  help  in  dealing  with  the  uncertainties  and  associated  risks  inherent  in  the  real 
problem.  A  risk-based  approach  can  overcome  some  major  pitfalls  by  adding  a  focus  on  the  multiplicity  of 
possible  outcomes  and  opening  up  the  possibility  of  richer  solutions  involving  portfolios  of  action  that 
produce  robustness  rather  than  narrow  optimality.  Portfolio-based  solutions  can  be  associated  with  cost- 
benefit  approaches,  but  this  has  not  been  common  in  practice. 

The  subject  of  C2  assessments  is  typically  both  highly  uncertain  and  opaque  in  nature.  The  team  should, 
therefore,  expect  a  complex  and  partly  hidden  set  of  risks  to  C2  studies  and  C2-related  decisions.  It  is 
recommended  that  serious  efforts  be  made  to  illuminate  the  risks,  and  it  is  good  practice  to  include  an 
explicit  risk-based  analysis  in  the  study  and  in  study  planning. 

Different  people  have  different  worldviews  and  different  approaches  to  risk  taking.  Analysts  should  seek 
to  find  out  about  how  risks  are  traded  for  expected  gain  by  the  study  sponsor  so  that  their  worldview  can 
be  appropriately  represented  in  the  assessment. 

Risk-based  analysis  needs  metrics  for  risks  and  failure  as  well  as  success  and  benefits,  which  means  that 
one  needs  a  way  of  expressing  various  levels  of  all  of  these  dimensions. 

In  C2  assessments,  analysts  need  to  be  particularly  alert  to  the  possibility  of  chaotic  behaviours  arising 
from  dynamic  interactions.  Human  and  organisational  factors  are  particularly  prone  to  this  type  of 
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instability.  A  sound  and  explicit  treatment  of  boundaries  and  system  definitions  during  problem 
formulation  is  essential  to  managing  this  aspect  of  the  assessment.  Holistic  systems  thinking  and 
complexity-based  analysis  may  be  needed  for  this  purpose. 

When  dealing  with  problems  involving  human  decisionmaking  the  analyst  must  be  aware  of  the  diversity 
of  courses  of  action  that  are  possible  as  the  situation  evolves.  The  analyst  must  ensure  that  these  courses  of 
action  are  represented  in  a  way  that  allows  for  the  possibility  of  a  discontinuous  set  of  possible  future 
states.  If  wide  divergence  in  course  of  action  selection  is  possible  over  the  timescales  under  consideration, 
then  the  treatment  of  scenarios  may  be  particularly  problematic  and  may  require  explicit  consideration 
under  the  risk  and  sensitivity  heading. 

10.5.3  Managing  Study  Risk 

C2  assessments  are  inherently  complex.  They  often  contain  poorly  understood  study  problems. 
These  factors  enhance  the  level  of  risk  in  the  design  and  conduct  of  the  assessment. 

It  is  good  practice  to  try  to  make  a  complete  list  of  risks  and  then  treat  them  in  appropriate  detail. 
A  top-down  approach  may  be  useful  in  assuring  a  certain  degree  of  completeness.  Independently  of  the 
depth  of  each  concrete  study,  it  is  good  practice  to  use  a  risk  perspective  to  explicitly  assess  the  robustness 
of  a  conclusion  or  recommendation.  If  possible,  one  should  try  to  keep  track  of  which  mechanisms 
underlie  each  risk,  the  probability  that  it  will  occur  and  how  it  can  be  mitigated,  then  consider  cost  of 
mitigation  and  cost  of  risk  impact,  before  taking  a  cost-benefit  approach  in  managing  the  study  risk  level. 

C2  problems  are  often  weakly  bounded.  There  is  a  particular  risk  associated  with  problem  formulation  and 
the  identification  of  factors.  Annex  F  lists  a  number  of  checklists,  which  have  proved  useful  in  ensuring  an 
adequately  complete  treatment  of  the  multiple  risk  areas. 

10.5.4  The  Generic  Risk  Register 

The  generic  risk  register  for  C2  assessment  (GRR)  is  a  good  starting  point  for  a  risk-based  analysis  of  a 
study  project  (not  the  decision  problem  supported  by  the  study).  The  lists  of  risks,  consequences  of 
impact,  and  mitigation  recommendations  are  directly  derived  from  the  COBP.  They  are,  therefore,  generic 
in  their  form  and  should  only  be  taken  as  a  starting  point  for  a  project-specific  risk  analysis.  The  GRR  has 
functionality,  which  allows  it  to  be  used  as  a  basic  tool,  or  the  list  of  risks  can  be  copied  into  a  more 
elaborate  tool  used  for  risk  handling  in  the  project. 

An  illustrative  example  is  the  case  study  undertaken  by  the  SAS-026  study  group,  when  a  brief  journey  of 
only  an  hour  through  the  generic  risk  register  turned  out  very  useful.  Although  the  study  team  was  well 
aware  of  the  advice  and  possible  pitfalls  in  advance,  explicitly  addressing  them  with  a  risk  perspective  led 
to  the  recognition  of  two  significant  flaws: 

•  The  low  number  of  planned  iteration  had  the  potential  to  lead  to  a  risk  of  an  inefficient  and 
unfocused  study  with  possibly  misleading  results;  and 

•  The  relatively  narrow  selection  of  methodological  approaches  entailed  a  risk  of  misleading 
conclusions.  There  could  be  important  consequences  of  varying  the  C2  system  that  were  not 
reflected  in  the  study,  and  the  possibly  biased  representation  would  represent  a  hidden  flaw  in 
conclusions. 

The  example  illustrated  both  the  usefulness  of  making  the  risk  assessment  explicit,  since  these  problems 
were  actually  known  to  all  participants  prior  to  the  risk  management  session,  and  also  illustrated  that  even 
surface  scratching  (as  was  the  case  here)  may  lead  to  significant  results.  It  is,  therefore,  advisable  not  to 
skip  risk  analysis  even  when  time  and  resources  are  limited. 
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10.6  COMMUNICATION  OF  RISK  AND  UNCERTAINTY 

The  high  level  of  uncertainty  (and  hence  risk)  in  C2  problems  and  their  assessment  mean  that  the 
communication  of  risk  and  uncertainty  to  study  customers,  sponsors,  and  stakeholders  is  of  particular 
importance.  The  value  of  a  quality  assessment  is  that  it  provides  decisionmakers  with  the  evidence  they 
need  to  make  better  decisions.  The  nature  and  quality  of  evidence  required  depend  upon  the 
decisionmaker’s  approach  to  and  tolerance  for  risk-taking  and  his/her  level  of  prior  knowledge  of  the 
problem  area  being  assessed. 

Communication  is  about  giving  the  receiver  of  a  message  a  right  impression,  not  about  formulating 
a  statement  that  is  formally  correct  on  its  own.  This  might  seem  obvious,  but  in  communicating 
uncertainties  and  risk,  it  should  be  given  particular  attention,  since  the  complexities  of  the  subject, 
human  limitations  in  reflecting  on  uncertainty,  and  the  lack  of  a  common  set  of  concepts  (and  also  hidden 
agendasjoften  will  make  communication  far  less  than  perfect. 

As  discussed  earlier,  some  uncertainty  can  be  reduced  by  analysis.  However,  some  uncertainty  is  inherent 
in  the  problem  and  needs  to  be  exposed  to  the  decisionmaker.  A  failure  to  do  this  can  lead  to  false 
confidence  in  study  conclusions.  C2  assessments,  in  particular,  will  contain  many  areas  of  unresolvable 
doubt  and  uncertainty.  These  should  be  openly  and  honestly  communicated  to  decisionmakers  to  avoid 
misinterpretation  of  study  conclusions.  Support  to  decisionmaking  under  uncertainty  is  a  vital 
complementary  activity  to  C2  assessment. 

Different  ways  of  framing  results  and  uncertainties  may  strongly  influence  the  way  results  are  perceived. 
This  should  be  considered  thoroughly  to  assure  compliance  with  ethical  standards.  One  should  be  aware 
that  stakeholders  (including  customers)  might  have  a  tendency  to  gloss  over  or  alternately  over-focus  on 
uncertainties.  An  analyst  should  take  care  in  communicating  an  objective  impression  of  risks  and 
uncertainty. 

Limitations  and  shortcomings  in  a  study  are  a  crucial  part  of  the  study  result  and  should  be  communicated 
as  effectively  as  possible.  This  enables  alignment  of  study  results  and  background  knowledge  on  the 
problem. 

Human  ability  to  understand  and  reason  on  uncertainty  is  limited  (Kahnmann  et  ah,  1982). 
These  limitations  should  be  given  particular  attention  when  communicating  risk  and  uncertainty  to 
stakeholders  and  decisionmakers. 

People  with  different  backgrounds  will  have  different  concepts  of  uncertainty  (e.g.  people  without  some 
mathematical  background  won’t  necessarily  intuitively  understand  Bayesian  concepts).  Thorough  dialogue 
may  be  needed  to  find  a  common  language.  Visualisation  techniques  will  be  helpful  in  this  regard  since 
they  are  usually  more  powerful  than  verbal  reference  to  abstract  concepts. 

One  should  be  careful  not  to  overwhelm  an  audience  with  details  on  uncertainties  and  possible 
shortcomings.  However,  a  continuing  dialogue  about  uncertainty  will  facilitate  a  common 
understanding.  Also,  the  analysis  team  should  be  aware  of  the  possibility  that  residual  uncertainties  may 
make  it  impossible  to  draw  robust  conclusions. 


10.7  CONCLUSIONS 

The  explicit  treatment  of  risk  and  uncertainty  is  best  practice  in  all  studies  and  is  of  particular  importance 
in  C2  assessment.  A  variety  of  dimensions  and  aspects  of  C2  assessments  carry  risk  and  uncertainty, 
particularly  because  they  are  liable  to  include  complex,  interacting  factors.  Even  when  study  resources  are 
limited,  it  is  best  practice  to  include  both  an  assessment  of  most  likely  outcome  (result),  to  do  sensitivity 
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analyses  looking  for  other  likely  outeomes,  and  to  take  a  risk-based  approach  looking  for  the  more 
extreme  possible  outcomes  (in  particular  failures). 

The  use  of  checklists  is  recommended  to  ensure  a  rigorous  treatment  of  risk  and  uncertainty.  The  best 
choice  of  checklist  depends  upon  personal  preference,  but  a  number  of  examples  are  presented  that  have 
been  found  useful  by  the  nations  contributing  to  this  COBP.  Additionally,  the  GRR  has  proved  useful  as 
an  aid  to  C2  study  risk  management. 


10.8  CHAPTER  10  ACRONYMS 


C2 

COBP 

DOTMLP 

DOTMLPF 

GRR 

METT-TC 

OOTW 

PESTLE 

RIGOUR 

ROE 

SWOT 


Command  and  Control 
Code  of  Best  Practice 

Doctrine,  Organisation,  Training,  Material,  Logistics,  Personnel 
Doctrine,  Organisation,  Training,  Material,  Leadership,  Personnel,  Facilities 
General  Risk  Register 

Mission,  Enemy,  Troops,  Terrain,  Troops,  Time,  and  Civil  considerations 
Operations  Other  Than  War 

Political,  Economic,  Social,  Technological,  Legal,  and  Environmental 

Repeatability,  Independence,  Grounding  in  reality.  Objectivity  of  process. 
Uncertainty,  and  Robustness 

Rules  of  Engagement 

Strengths,  Weaknesses,  Opportunities,  Threats 
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“We  are  what  we  repeatedly  do.  Excellence,  then,  is  not  an  act,  but  a  habit.  ”  -Aristotle 

PRODUCTS 

The  purpose  of  study  products  is  threefold:  to  communicate  results  to  sponsors  and  stakeholders; 
to  provide  a  lasting  record  of  what  went  into  the  planning;  and  to  establish  credibility  within  the  technical 
community.  Verbal  communication  and  progress  reports  may  be  necessary,  especially  for  a  short  study, 
however  a  thorough  written  record  is  essential  to  the  credibility  and  longevity  of  study  results. 

Study  products  that  are  delivered  to  the  customer  generally  include  a  study  plan,  periodic  status/progress 
reports,  and  a  final  report.  Several  other  products  may  be  produced  and  delivered  to  the  customer. 
Some  study  products  are  created  and  maintained  primarily  for  internal  study  support.  These  products  are 
not  unique  to  C2  OOTW  studies  or  to  general  C2  studies. 
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Products  that  are  typically  produced  from  a  study  include: 


11.1  STUDY  PLAN 

The  study  plan  described  here  is  a  subset  of  the  Study  Management  Plan  of  section  4-E.  It  generally 
includes  the  initial  problem  formulation  and  solution  strategy,  emphasizing  a  general  understanding  of  the 
problem,  deliverables,  budget,  time  line,  and  solution  approach.  The  study  plan  includes  at  a  minimum: 

•  Statement  of  the  problem  (problem  formulation)  -  “the  what”; 

•  Solution  strategy  “the  how  and  the  when”; 

•  Tasks  and  their  relationships;  and 

•  Milestones. 

11.2  PERIODIC  STATUS/PROGRESS  REPORTS 

Periodic  status  reports  describe  the  activities  of  the  most  recent  period  and  the  expected  activities  of  the 
next  period,  and  link  the  activities  to  the  tasking  statement.  Status  reports  also  contain  cost  information 
and  track  adherence  to  the  planned  schedule.  One  of  the  most  important  sections  of  the  status  report  is 
the  “Problems  Encountered”  section.  This  section  should  include  technical  problems,  budgetary  problems, 
and  most  importantly  problems  relating  to  sponsor/assessment  team  relations.  These  reports  may  be 
delivered: 

•  To  the  sponsor  and  other  stakeholders;  and 

•  To  the  peer  review  team. 

11.3  FINAL  REPORT 

The  final  report  contains  sections  that  address  the  following: 

•  Objectives  (customer  question  and  the  problem  formulation); 

•  Scope  and  assumptions; 

•  Approach  (solution  strategy); 

•  Findings/conclusions  (with  caveats); 

•  Recommendations  (optional); 

•  Future  challenges  (optional); 

•  Appendices; 

•  Data  collection  instruments  or  discussion  of  instrumentation  (optional); 

•  Data  dictionary  (optional); 

•  Data  (optional); 

•  Glossary  of  terms/acronyms;  and 

•  References. 

The  findings/conclusions  and  recommendations  (if  present)  should  address  each  of  the  objectives. 
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The  final  report  should  be  produced  as  an  archivable  document  that  can  be  readily  accessed  by  the 
community.  This  means  that  the  document  is  produced  in  electronic  form  using  commercial  standards. 
Ideally  these  documents  should  be  made  available  through  web  sites.  However,  it  is  recognised  that 
security  considerations  and  language  will  limit  the  availability  of  many  documents.  Briefings  are  a  useful 
form  of  communication.  It  is  desirable  that  a  briefing  accompany  the  more  formal  final  report.  However, 
when  only  a  briefing  is  produced,  it  must  be  annotated. 

Command  and  control  (C2)  problems  tend  to  be  multi-dimensional  and  highly  complex.  Therefore, 
they  pose  unique  challenges  to  the  assessment  team  in  communicating  the  result  effectively  to  the 
decisionmaker.  There  are  several  steps  that  can  be  taken  to  facilitate  this  communication.  First,  if  the 
product  is  an  architecture,  it  is  recommended  that  the  templates  developed  by  community  to  depict 
alternative  architectural  perspectives  (e.g.  operational,  system,  and  technical  architectures)  be  employed. 
Second,  it  has  proven  useful  to  employ  “stop  lighf’  charts  (red,  yellow,  and  green)  to  characterise 
measures  of  performance/effectiveness.  However,  such  techniques  are  useful  only  in  conveying  qualitative 
insights.  A  tendency  to  rely  heavily  upon  this  presentation  technique  may  result  in  assessments  that  do  not 
drive  to  quantitative  results.  Third,  there  is  a  need  to  develop  and  employ  visualisation  techniques  to 
capture  the  full  richness  of  the  insights,  particularly  risk  and  uncertainty  (e.g.  depicts  the  distribution 
rather  than  just  the  statistical)  that  are  derived  in  assessments.  Preliminary  research  is  underway  in  this 
area.  It  needs  to  be  extended  and  translated  into  application. 

Because  C2  data  are  rare,  every  effort  should  be  made  to  retain  the  data  and  make  it  available  to  other 
recorders.  Sometimes  others  will  require  “sanitising”  the  data  to  prevent  anyone  knowing  which  units  or 
exercises  produced  it.  When  archiving  data  the  metadata  labels  that  identify  the  conditions  under  which  it 
was  collected  should  also  be  preserved. 

A  peer  review  process  is  an  essential  part  of  producing  a  final  report.  It  should  begin  with  the  preparation 
of  the  study  plan  and  continue  throughout  the  life  of  the  assessment.  Draft  products  must  be  provided  to 
reviewers  in  ample  time  for  them  to  review  and  comment  on  the  product  and  for  the  team  to  reflect  their 
comments.  A  failure  to  institute  an  adequate  review  process  can  compromise  the  quality,  credibility, 
and  utility  of  the  assessment. 


11.4  OTHER  DELIVERED  PRODUCTS 

Several  products  may  be  created  and  delivered,  depending  on  the  needs  of  the  project: 

•  Description  of  the  assessment  participants  (including  assessment  team)  and  their  relationships 
(defined  in  2-B); 

•  Description  of  the  budget  and  time  constraints  to  provide  context  information  for  future  studies; 

•  Human  and  Organisational  Factors  Checklist; 

•  Scenario  details; 

•  Video  and  audio  presentations; 

•  Created  models,  spreadsheets,  decision  support  tools,  etc.; 

•  Simple  models  and  tools  (EG  for  the  sponsor  and  other  parties  to  interactively  explore 
interrelationships  between  the  variables  of  the  study);  and 

•  Experimentation  Campaign  Plan  (if  the  C2  Assessment  makes  use  of  a  series  of  linked  events 
such  as  seminars,  wargames,  command  post  exercises  (CPX),  field  training  exercises  (FTX)  etc.). 
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1 1.5  OTHER  PRODUCTS 

In  addition  to  study  products  that  are  delivered  to  the  sponsor  there  are  a  number  of  products  that  best 
practice  demands  are  produced  and  maintained  during  the  course  of  a  study.  These  include: 

•  A  project  journal; 

•  Study  management  plan  (defined  in  4-E); 

•  Data  collection  plan; 

•  Data  analysis  plan;  and 

•  Study  glossary. 

11.6  CONCLUSION 

A  study  is  generally  appraised  based  on  the  quality  of  its  study  products.  This  Code  of  Best  Practice  aims 
to  highlight  important  areas  that  will  improve  both  the  assessment  process,  and  the  quality,  longevity, 
and  utility  of  the  study  products.  The  goal  is  to  make  the  state  of  practice  one  and  the  same  with  the  state 
of  the  art. 

“Of  all  the  communities  available  to  us  there  is  not  one  I  would  want  to  devote  myself  to,  except  for  the 
society  of  the  true  searches,  which  has  very  few  living  members  at  any  time.  .  .  “  -Albert  Einstein 
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A 

ACCES 

Army  Command  and  Control  Evaluation  System 

ACE  Resources 

Allied  Command  Europe  Resources  (part  of  SHAPE) 

AF(N) 

Regional  Command 

AHWG 

AC/243  Panel  7  Ad  Hoc  Working  Group 

AI 

Artificial  Intelligence 

AMF(E) 

ACE  Mobile  Force  (Eand) 

ANT 

Actor  Network  Theory 

ARRC 

ACE  Rapid  Reaction  Corps 

ATS 

Norman’s  Activation  Trigger  Scheme 

B 


C 

C2 

C3 

C3I 

C4 

C4I 

C4ISR 

CIMIC 

CIS 

CISS 

COBP 

CPX 

CTA 


Command  and  Control 

Command,  Control,  and  Communications  or  Consultation 

Command,  Control,  Communications  or  Consultation,  and  Intelligence 

Command,  Control,  Communications  or  Consultation,  and  Computers 

Command,  Control,  Communications  or  Consultation,  Computers,  and  Intelligence 

Command,  Control,  Communications  or  Consultation,  Computers,  Intelligence, 

and  Surveillance  and  Reconnaissance 

Civil-Military  Co-operation 

Command  Information  Systems 

Center  for  Information  Systems  Security 

Code  of  Best  Practice 

Command  Post  Exercise 

Constructive  Technology  Assessment 


D 

DCEP 

DISA 

DOTMEP 

DOTMEPF 

DP 

DSTE 


Data  Collection/Engineering  Plan 
Defence  Information  Systems  Agency 

Doctrine,  Organisation,  Training,  Material,  Eeadership,  Personnel 

Doctrine,  Organisation,  Training,  Material,  Eeadership,  Personnel,  Facilities/Forces 

Dimensional  Parameters 

Defence  Science  and  Technology  Eaboratories  (UK) 


E 

EEA  Essential  Elements  of  Analysis 
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F 

FS 

FTX 


G 

GRR 


H 

HB(A) 

HEAT 

HQ 

I 

10 

IRDS 

IRTF(L) 

J 

JCSC 

JCSE 

JFCOM 


K,L 

M 

MAPEX 

MAUT 

MCES 

M-E-M 

METT-TC 

MND(C) 

MoCE 

MoE 

MoFE 

MoM 

MoP 

MoPE 

MORS 

M&S 

M-T-M 


ORGAmZATION 


Feasibility  Study 
Field  Post  Exereise 


Generie  Risk  Register 


UK  Historieal  Braneh  (Army) 

Headquarters  Effectiveness  Assessment  System 
Headquarters 


International  Organisation 
Information  Resource  Dictionary  Systems 
Immediate  Reaction  Task  Force  (Eand) 


Joint  Sub-Regional  Command  South  Centre 
Joint  Sub-Regional  Command  South  East 
Joint  Forces  Command 


Map  Exercise 

Multi-Attribute  Utility  Theory 

Modular  Command  and  Control  Evaluation  Structure 

Model-Exercise-Model 

Mission,  Enemy,  Troops,  Terrain,  Troops,  Time,  and  Civil  considerations 
Multinational  Division  (Centre) 

Measures  of  C2  Effectiveness 
Measures  of  Effectiveness 
Measures  of  Force  Effectiveness 
Measures  of  Merit 
Measures  of  Performance 
Measures  of  Policy  Effectiveness 
Military  Operations  Research  Society 
Modelling  and  Simulation 
Model-Test-Model 
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N 

NATO 
NCSA 
NGO 
NL  MOD 

o 

OA 

OOTW 

OR 

P 

PESTLE 

PfP 

PRL 

PVO 

Q 

R 

RIGOUR 

ROE 

S 

SACLANT  OA 

SCOT 

SFS 

SHADE 

SHAPE 

SMP 

SOW 

STA 

STAFFEX 

STS 

SWOT 

T 

TA 

TRAC 

TRADOC 

TTP 


North  Atlantic  Treaty  Organisation 

NATO  C3  (Consultation,  Command  &  Control)  Agency 

Non-Govemmental  Organisations 

Netherlands  Ministry  of  Defense 


Operational  Analysis 
Operations  Other  Than  War 
Operations  Research 


Political,  Economic,  Social,  Technological,  Legal,  and  Environmental 
Partnership  for  Peace 
Policy  Requirements  Land 
Private  Volunteer  Organisations 


Repeatability,  Independence,  Grounding  in  reality.  Objectivity  of  Process, 
Uncertainty,  and  Robustness 
Rules  of  Engagement 


Supreme  Allied  Command  Atlantic  Operational  Analysis  Cell 

Social  Construction  of  Technology 

Strike  Force  South 

Shared  Data  Environment 

Supreme  HQ  Allied  Powers  Europe 

Study  Management  Plan 

Statements  of  Work 

Surveillance  Targeting  and  Acquisition 

Staff  Exercise 

Science  and  Technology  Studies 
Strengths,  Weaknesses,  Opportunities,  Threats 


Technology  Assessment 
TRADOC  Analysis  Center 
US  Army  Training  &  Doctrine  Command 
Tactics,  Techniques,  and  Procedures 
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U 

V 

VTC 

VV&A 

vv&c 

w 

WBS 

X,  Y,Z 


ORGAmZATIOM 


Video  Teleconference 

Verification,  Validation,  and  Accreditation 

Verification,  Validation,  and  Certification 


Work  Breakdown  Structure 
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The  MORS  Code  of  Ethies  and  Responsibilities  for  Praetitioners  of  Military  Operations  Researeh 


Military  OR  Professionals  must  aspire  to  be: 

•  Honest,  open  and  trustworthy  in  all  their  relationships; 

•  Reliable  and  eonsistent  in  the  eonduet  of  assignments  and  responsibilities,  always  doing  what  is 
right  rather  then  expedient; 

•  Objective,  constructive  and  accurate  in  what  they  say  and  write; 

•  Accountable  for  what  they  do  and  choose  not  to  do; 

•  Respectful  of  the  work  of  others,  giving  due  credit  and  refraining  from  criticism  of  them  unless 
warranted;  and 

•  Free  from  affiliation  with  others  or  with  activities  that  would  compromise  them,  their  employers, 
or  the  Society. 
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This  checklist  presents  a  summary  of  a  particular  set  of  human  and  organisational  issues  that  is 
specifically  relevant  to  behaviour  and  performance  of  humans  in  command  and  control  situations. 
Its  purpose  is  to  sensitise  the  analyst  and  help  him  to  assess  whether  human  or  organisational  issues  are 
part  of  the  problem  domain  and  should  be  addressed  in  the  solution  space.  Also,  it  should  assist  him  in 
identifying  human  sciences  disciplines  for  consultation. 

It  should  be  pointed  out,  however,  that  this  checklist  is  preliminary  and  must  not  be  considered  to  be 
comprehensive.  The  field  of  human  sciences  is  too  large  and  the  set  of  human  and  organisational  issues  to 
ill  defined  to  provide  the  analyst  with  an  exhaustive  checklist,  at  this  time.  Research  in  this  area  is  ongoing 
and  we  expect  that  later  editions  of  this  document  will  provide  the  analyst  revised  and  improved 
checklists. 


1.0  HUMAN  ISSUES 

Physiological  Factors 

This  refers  to  bio-medical  and  environmental  factors  that  influence  behaviour  and  performance. 
Stress: 

•  Fatigue  and  lack  of  sleep; 

•  Blood  sugar; 

•  Fitness  conditions; 

•  Weather  conditions;  and 

•  Geographical  terrain  conditions. 

Ergonomic  Factors  (e.g.  performance  degradation  due  to  working  in  protective  suits) 
Ergonomics  is  a  science  discipline  that  studies  human  work  and  work  environment  relationships. 

Behavioural  Factors  (related  to  functioning  in  a  C2  group) 

•  Social  Competence  &  Experience; 

•  Capability  to  interact  with  others; 

•  Communication  skills; 

•  Language  skills; 

•  Empathy  (Social  awareness  /  understanding); 

•  Conflict  handling  style; 

•  Frustration  handling  style; 

•  Military  Competence  &  Experience; 

•  Handling  danger; 

•  Defining  mission  objectives;  and 

•  Task  Competence  &  Experience. 
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Cognitive  Factors 

These  factors  are  related  to  how  humans  perceive  their  environment,  how  they  give  meaning  to  what  they 
see. 


•  Information  processing  style; 

•  Information  processing  capacity; 

•  Creating  situational  awareness; 

•  Individual  decisionmaking; 

•  Risk  tolerance; 

•  Pre-disposition!; 

•  Receptivity  to  new  information  (open  /  closed); 

•  Expertise;  and 

•  Prior  training  and  knowledge  (operational  codes). 

Emotional  Factors 

•  Morale; 

•  Attitude; 

•  Separation  from  family; 

•  Fear; 

•  Stress;  and 

•  Resilience  (ability  to  overcome  negative  feedback). 

Leadership  Factors 

There  exists  no  clear-cut  concept  of  leadership  effectiveness,  but  we  do  know  some  of  the  factors  that 
affect  the  effectiveness  of  a  leader. 

•  Expectation  and  behaviour  of  superiors; 

•  Expectations  and  behaviour  of  subordinates; 

•  Expectations  and  behaviour  of  colleagues; 

•  Personality  and  experience; 

•  Organisational  culture  and  policy  -  with  regard  to  leadership  and  command.  Allocation  or 

responsibility  and  authority; 

•  Ability  to  motivate  and  to  direct  others; 

•  Moral  and  judicial  responsibilities;  and 

•  Coaching  capabilities  (towards  subordinates). 


2.0  ORGANISATIONAL  ISSUES 

Structure 

•  Number  of  echelons  or  layers; 

•  Span  of  control  for  nodes; 
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•  Pattern  of  linkages  between  nodes  (e.g.  hierarchical,  multi- connected); 

•  Permanent  versus  transitory;  and 

•  Formal  versus  informal. 

Function 

•  Distribution  of  responsibility; 

•  Distribution  of  authority; 

•  Distribution  of  information; 

•  Functional  specificity;  and 

•  Ambiguity  in  command  relationships. 

Capacity 

•  Differences  in  communication  systems  /  architectures; 

•  Differences  in  information  processing  systems  /  architectures; 

•  Differences  in  field  training  and  operational  experience; 

•  Differences  in  personnel; 

•  Experience; 

•  Training;  and 

•  Cognitive  ability. 


Roles 

•  Allocation  of  responsibility  /  authority;  and 

•  Role  conflicts.  The  interference  of  multiple  roles  in  one  individual. 

Sociological  Factors 

•  Understanding  of  environment; 

•  Political; 

•  Social;  and 

•  Cultural. 

Interoperability  Issues 

•  National  and  cultural  differences; 

•  Organisational  approaches  and  values; 

•  Communication  standards  and  technology;  and 

•  Differences  in  perceptions. 

Organisational  Command  Style 

•  Decentralised  or  centralised; 

•  Collaborative  versus  authoritative; 
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•  Formal  versus  informal;  and 

•  Command  products:  orders,  objectives,  missions. 

Organisational  Culture 

Shapes  patterns  of  organisational  behaviour  and  reflects  thought  and  activity  patterns  by  members  of  the 
organisation. 

•  Belief  systems; 

•  Organisational  norms  &  how  are  they  expressed; 

•  Organisational  values  &  how  are  they  expressed;  and 

•  Open  to  organisational  learning. 

i.e.  Does  the  organisational  culture  embed  characteristics  that  enable  and  facilitate 
organisational  earning? 

3.0  INTERACTION  BETWEEN  HUMAN  AND  ORGANISATIONAL  ISSUES 

Group  Decisionmaking 

Groups  can  exert  social  pressure  on  members  that  affect  the  decisionmaking  capability  of  individual 
members. 

Group  Dynamics 

The  behaviour  of  individuals  in  a  group  will  be  influenced  by  the  fact  that  they  are  interacting  in  a  group. 
This  interactive  process  contains  a  number  of  dynamic  dimensions  and  does  affect  the  result  of 
interaction.  A  command  post  is  a  group  of  people  and  its  performance  is  affected  by  it  group  dynamics. 

•  Social  interaction  &  communication; 

•  Communication; 

•  Social  identity  and  social  conflicts; 

•  Individual  dominance/leadership; 

•  Cohesion;  and 

•  Teambuilding,  teamwork. 


Trust  in  group  member’s  competence  and  loyalty. 

Cultural  Factors 

Culture  is  more  or  less  the  whole  of  beliefs  and  assumptions,  including  norms  and  values,  about  things 
and  behaviour  in  a  group.  As  such  it  guides  the  behaviour  of  people,  also  in  the  military  and  in  command 
posts. 

Note!  Culture  itself  is  immaterial,  intangible.  It  can  only  be  distilled  form  observed  actions  by 
members  of  an  organisation  or  societal  group. 

Note!  Culture  is  learned  through  interaction  with  others. 

•  Especially  in  multi-national  OOTW,  members  of  different  cultural  background  have  to 
interact.  Mutually  acceptable  norms  of  behaviour  will  have  to  be  developed  in  the 
process. 

•  Socialisation  process. 
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Culture  Lag 

Changes  in  the  environment  or  to  the  organisation  often  require  adaptations  in  behaviour. 
Sometimes  these  changes  are  required  or  introduced  so  fast  that  an  organisation  hasn ’t  had  the 
time  to  the  whole  of  its  culture  to  adapt  to  the  new  situation,  i.e.,  to  embed  the  changes  in  its 
culture.  This  may  lead  to  friction. 

•  E.g.l:  The  introduction  of  new  technologies  may  change  the  way  we  work.  If  this  is  done 
too  fast,  or  without  thoughtful  guidance  it  may  have  a  disruptive  effect  on  behaviour  and 
thus  on  organisational  effectiveness;  and 

•  E.g.2:  Peace  Keeping  and  War  Operations  each  require  different  types  of  behaviours 
which  are  often  quite  different  from  the  type  of  behaviour  in  the  peaceful  environment  at 
home.  Western  armies  operate  on  personnel  rotation  schedules  whereby  individuals 
change  frequently  from  one  situation  to  another.  This  may  also  result  in  culture  lag 
causing  friction. 

Sub-Culture 

Within  the  larger  set  of  culture,  groups  may  develop  specialised  cultures,  such  as,  e.g.,  distinct 
unit  cultures. 

•  Sometimes  these  sub-cultures  may  conflict  over  some  issues  with  the  dominant 
organisational  or  societal  culture.  Understanding  these  differences  is  often  the  first  step  in 
resolving  these  situations. 

Social  Control 

Individual  (group,  organisational)  members  watch  and  correct  each  other  if  they  accepted 
procedures  of  behaviour  aren  ’t  followed. 

•  If  this  corrective  action  pattern  is  too  strict,  the  group  becomes  rigid  and  is  less  able  to 
adapt  to  new  situations;  and 

•  If  it  is  too  loose,  and  no  culture  or  norms  are  enforced,  chaos  and  uncertainty  may  rule  the 
group. 

Commanders  Intent 

A  leader  can  affect  the  current  (sub)  culture  of  his  unit  by  setting  the  example  and  by  establishing 
and  enforcing  required  procedures  and  behaviour. 

•  Both  must  be  synchronized,  else  friction  will  occur;  and 

•  By  action  and  word  a  leader  sets  the  level  and  direction  of  (social)  control. 


Ethics 

Cooperability 

•  Effectiveness  of  communication;  and 

•  Willingness  to  co-operate  and  collaborate. 


4.0  ENVIRONMENTAL  FACTORS 

These  factors  affect  individual,  organisational  and  group  issue. 

•  Noise; 

•  Visibility; 
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•  Temperature  /  humidity; 

•  Terrain  type; 

•  Infrastrueture  in  area  of  operations,  (e.g.  transport,  eommunieation,  healtheare,  agrieulture,  food 
distribution,  water,  eivil  administration-infrastruetures); 

•  Military  support  infrastrueture; 

•  Soeial,  eeonomie,  and  politieal  situation  in  area  of  operations; 

•  Rules  of  engagement; 

•  Ease  of  interaetion; 

•  Command  post  layout; 

•  Communieation  meehanisms  (e.g.  voiee,  teletype,  VTC);  and 

•  Geographie  distribution  of  the  eommand. 


5.0  RELATED  HUMAN  SCIENCE  DISCIPLINES 

This  list  provides  an  overview  of  the  relevant  seientifie  disciplines  with  the  expertise  to  answer  questions 
arising  in  the  context  of  human  and  organizational  issues  in  command  and  control.  The  list  implies  a 
rough  taxonomy  of  disciplines  only.  Even  though  boundaries  between  disciplines  are  frequently  frizzy, 
an  attempt  was  made  to  minimise  the  overlap. 

1  Psychology 

1 . 1  Individual  Psychology 

1.1.1  Cognitive  Psychology 

1.1.2  Eearning  Psychology 

1.1.3  Action  Theories 

1.1.4  Differential  Psychology 

1.1.5  Psychometrics 

1.2  Social  Psychology 

1.3  Clinical  Psychology 

2  Educational  Sciences  and  Pedagogics 

3  Sociology 

3.1  Social  Morphology  (e.g.  Social  Stratification,  Demography) 

3.2  Sociology  of  Political  Power  (See  Also  Political  Sciences) 

3.3  Mass  Communication 

3.4  Sociological  Methodology  (Empirical  Social  Research) 

4  Organisational  Sciences 

5  Economic  Sciences 

6  Political  Sciences 

6. 1  Science  of  Domestic  Politics 

6.2  Science  of  International  Relations 

6.3  Political  Disaster  Research 


D-6 


RTO-TR-081 


ANNEX  D  -  HUMAN  AND  ORGANISATIONAL  ISSUES  CHECKLIST 


7  Ergonomics,  Human  Factors  Research 

7. 1  Physiology  and  Anthropometries 

7.2  Technieal  Ergonomies  (Man-Maehine-Interfaee,  Workspaee) 

7.3  Cognitive  Ergonomics  (See  also  Individual  Psychology) 

7.4  Crew  Ergonomics  (See  also  Social  Psychology) 

7.5  Safety  and  Health  Hazard  Prevention 

8  International  Law  of  War,  International  Humanitarian  Law 

9  Ethics 

9. 1  Deontology  (Theories  of  Ethical  Norms  and  Values) 

9.2  Theories  of  Ethical  Practise  and  Morality 

10  Social  and  Legal  Philosophy 

11  Cultural  Anthropology 
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A  number  of  different  definitions  of  risk  and  uncertainty  are  in  common  use,  and  this  can  lead 
to  confusion  and  misunderstanding.  This  COBP  adopts  specific  working  definitions  of  risk,  uncertainty, 
and  sensitivity;  however,  this  annex  lists  a  range  of  others  for  information. 


1.0  RISK 

Risk  is  defined  in  this  COBP  as  the  possibility  of  suffering  harm  or  loss.  In  other  words,  exposure  to  harm 
or  loss.  This  includes  an  opportunity  loss. 

Alternate  definitions  of  risk  in  common  use  include: 

•  Not  achieving  your  objective; 

•  The  likelihood  of  not  achieving  your  objective; 

•  A  threat  to  successful  outcome; 

•  An  assessment  of  the  probability  of  failure; 

•  An  uncertain  future  scenario;  and 

•  A  perception  of  consequential  pain. 

The  alternate  definitions  may  be  seen  as  subtle  variations  of  each  other  and  are  all  equally  valid.  However, 
this  very  subtlety  can  be  a  source  of  confusion  and  doubt  within  a  study  unless  a  clear  working  definition 
is  adopted. 


2.0  UNCERTAINTY 

Uncertainty  is  defined  in  this  COBP  as  an  inability  to  determine  a  variable  value  or  system  state 
(or  nature)  or  to  predict  its  future  evolution. 

Alternate  definitions  of  uncertainty  in  common  use  include: 

•  A  lack  of  clarity  in  the  definition  of  a  system  or  variable; 

•  A  lack  of  confidence  in  an  assumption  or  result;  and 

•  A  lack  of  knowledge  about  a  subject  of  interest. 

All  of  these  definitions  have  some  validity,  but  this  should  not  be  allowed  to  cloud  the  thinking  during  C2 
assessment. 

3.0  SENSITIVITY 

Sensitivity,  as  used  in  this  COBP,  describes  a  pre-disposition  to  respond  strongly  to  a  stimulus  or 
variations  in  an  input  factor. 
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Alternate  definitions  for  sensitivity  in  common  use  include: 

•  An  indication  of  the  importance  or  criticality  of  a  feature  or  variable  of  an  analysis;  and 

•  A  measure  of  the  political  profile  of  the  subject  of  study. 

Sensitivity,  as  it  relates  to  the  production  of  robust  analysis,  must  be  clearly  defined  in  terms  of  response 
to  stimulus  rather  than  importance  or  criticality.  In  C2  systems  many  insensitive  factors  are  critical  and 
many  sensitive  ones  are  relatively  unimportant. 
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This  revised  and  expanded  version  of  the  COBP  for  C2  assessment  was  developed  by  SAS-026  building 
upon  the  initial  (1998)  version  of  the  COBP  produeed  by  SAS-002.  This  edition  of  the  COBP  is  a 
synthesis  of  deeades  of  expertise  from  various  eountries  and  hundreds  of  analyses.  It  was  developed  using 
a  set  of  case  studies  and  incorporates  feedback  from  users  of  the  initial  version.  Lastly,  SAS-039  provided 
a  peer  review  of  the  final  draft  product.  Members  of  SAS-026  are  listed  below. 

Dave  Alberts  (chair),  United  States 
Tim  Bailey,  United  States 
Paul  Chouinard,  NCSA 
Cornelius  d’Huy,  The  Netherlands 
Uwe  Dompke,  NCSA 
Dean  Hartley,  United  States 
Richard  Hayes,  United  States 
Reiner  Huber,  Germany 
Don  Kroening,  United  States 
Stef  Kurstjens,  The  Netherlands 
Nick  Lambert,  NCSA 
Georges  Lascar,  France 
Christian  Manac’h,  France 
Graham  Mathieson,  United  Kingdom 
Jim  Moffat,  United  Kingdom 
Orhun  Molyer,  Turkey 
Valdur  Pille,  Canada 
David  Signori,  United  States 
Mark  Sinclair,  United  States 
Mink  Spaans,  The  Netherlands 
Stuart  Starr,  United  States 
Swen  Stoop,  The  Netherlands 
Hans  Olav  Sundfor,  Norway 
Klaus  Titze,  Germany 
Andreas  Tolk,  United  States 
Corinne  Wallshein,  United  States 
Gary  Wheatley,  United  States 
John  Wilder,  United  States 
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1.1  BACKGROUND  AND  AIMS 

NATO  has  produced  a  Code  of  Best  Practice  (COBP)'  in  order  to  facilitate  high  quality  assessment  in  the 
area  of  Command  and  Control  (C2).  The  COBP  is  the  product  of  international  collaboration  drawing 
together  the  operational  and  analytical  experience  of  leading  military  and  civilian  defence  experts  from 
across  the  NATO  nations.  The  COBP  enhances  the  understanding  of  best  practice  and  outlines  a  structured 
process  for  the  conduct  of  operational  analysis  for  C2,  which  is  the  core  capability  of  Information  Age 
defence  and  security. 

The  command  and  control  aspects  of  military  capability  are  difficult  to  assess.  Use  of  the  COBP  will 
increase  the  likelihood  of  quality  products  that  are  complete,  relevant,  transparent,  credible  and 
authoritative.  In  particular,  the  COBP  will: 

•  Support  broadening  of  the  analysis  process; 

•  Support  effective  use  of  analysis  in  direct  support  of  operations; 

•  Improve  the  quality  and  coherence  of  business  cases; 

•  Reduce  risk  and  cost  of  the  planning,  preparation,  analysis  and  presentation  of  supporting 
documentation;  and 

•  Provide  a  methodology  acceptable  to  both  analysts  and  decisionmakers. 

The  COBP  facilitates  effective  structuring  of  the  analytic  process.  It  describes  a  framework  that  can  be 
used  to  design,  execute,  review  and  deliver  high  quality  C2  assessments  that  include  all  key  factors  and 
stakeholders.  Use  of  the  COBP  should  be  regarded  as  a  community  standard  for  all  levels  and  scales  of 
assessment  studies. 


1.2  DEFINITION  OF  C2 

C2  has  been  defined  by  NATO  as  Military  Function  01:  “The  Organisation,  Process,  Procedures, 
and  Systems  necessary  to  allow  timely  political  and  military  decisionmaking  and  to  enable  military 
commanders  to  direct  and  control  military  forces^.”  C2  systems  are  further  defined  in  NATO  documents 
to  include:  headquarters  facilities,  communications,  information  systems,  and  sensors  &  warning 
installations^.  More  recently,  the  term  “C2”  has  referred  to  the  collaborative  and  consultative  processes 
that  are  an  inherent  part  of  coalition  operations. 

For  the  purposes  of  the  COBP,  the  term  C2  is  intended  to  be  an  umbrella  term  that  encompasses  the 
concepts,  issues,  organisations,  activities,  processes,  and  systems  associated  with  the  NATO  definition  of 
C2  as  well  as  the  other  terms  enumerated  above. 


*  North  Atlantic  Treaty  Organisation  (NATO)  NATO  Code  of  Best  Practice  for  C2  Assessment,  2002. 

^  North  Atlantic  Treaty  Organisation  (NATO)  Annex  B  to  MC  Guidance  for  Defence  Planning,  MC-299/5  (1996). 

^  North  Atlantic  Treaty  Organisation  (NATO)  Bi-MNC  C2  plan  part  2  -  Command  and  Control  Requirements  (1998). 
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1.3  WHY  IS  C2  SPECIAL? 

C2  is  special  because  it  explicitly  involves  representation  of  the  human  component.  The  focus  of  military 
research  and  analysis  has  predominantly  been  on  the  physical  domain.  C2  deals  with  distributed  teams  of 
humans  operating  under  stress  and  in  a  variety  of  other  operating  conditions.  C2  problems  are  thus 
dominated  by  their  information,  behavioural,  and  cognitive  aspects  that  have  been  less  well  researched  and 
understood.  This  focus  creates  a  multidimensional,  complex  analytic  space  that  involves  multi-sided 
dynamics  including  friendly,  adversary,  and  other  actors,  action-reaction  dynamics,  and  tightly  coupled 
interactions  among  elements  such  as  doctrine,  concepts  of  operations,  training,  materiel,  and  personnel. 

C2  issues  are  difficult  to  decompose  and  recompose  without  committing  errors  of  logic.  Moreover, 
the  composition  rules  by  which  the  various  factors  inherent  to  C2  interact  are  poorly  understood  except  in 
arenas  that  have  been  previously  studied  in  detail.  Finally,  the  C2  arena  is  weakly  bounded,  with  issues 
that,  although  on  initial  examination  appear  quite  finite,  often  prove  to  be  linked  to  very  high-level  factors. 

The  COBP  is  intended  to  assist  the  community  in  dealing  with,  and  overcoming,  the  barriers  to  effective 
C2  assessment. 

1.4  STRUCTURE  OF  THE  GUIDE 

The  rest  of  this  guide  is  structured  to  follow  the  chapters  of  the  COBP  and  abstracts  definitions,  principles 
and  key  points  of  guidance. 
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Chapter  2  -  PREPARING  FOR  SUCCESS:  ASSESSMENT 
PARTICIPANTS,  RELATIONSHIPS,  AND  DYNAMICS 


The  issues  assoeiated  with  assessment  partieipants,  relationships,  and  dynamies  must  be  addressed  in 
order  to  establish  a  sound  foundation  for  a  successful  assessment.  As  an  initial  step,  the  individuals  who 
are  involved  in  the  study  must  be  identified,  the  relationships  among  the  participants  must  be  understood, 
and  a  common  understanding  of  the  study’s  goals,  objectives,  scope,  and  administrative  aspects 
(e.g.,  schedule,  budget)  must  be  established  and  documented.  An  assessment  team  must  be  assembled  that 
manifests  the  necessary  skills  and  experience,  and  the  broad  plan  of  attack  must  be  formulated. 


2.1  ASSESSMENT  PARTICIPANTS 

Early  in  the  study,  the  assessment  team  should  identify  which  individuals  and  organisations  should 
perform  key  roles  in  the  assessment.  The  roles  of  interest  include,  inter  alia,  assessment  team  members, 
decisionmakers  or  problem  owners,  stakeholders,  bill  payers,  existing  and  future  study  teams, 
peer  reviewers,  data  providers,  assumption  providers,  and  data  collectors.  It  is  prudent  for  the  assessment 
team  to  map  the  roles  onto  the  individuals  and  organisations  involved  and  to  understand  their 
interrelationships. 


2.2  RELATIONSHIPS  AMONG  PARTICIPANTS  AND  THE  CONDUCT  OF 
THE  ASSESSMENT 

The  relationship  among  the  assessment  team,  the  key  sponsor,  and  the  stakeholders  is  of  paramount 
importance  and  will  influence  the  course  and  success  of  the  effort.  The  assessment  team  should  be  aware 
that  the  diverse  participants  may  have  divergent  perspectives  and  agendas. 

The  assessment  team  should  undertake  the  following  actions: 

•  Maintain  long-term  relationships  (including  an  ongoing  dialogue)  with  the  sponsor  and 
stakeholder  organisations.  This  will  yield  substantial  dividends  in  the  form  of  easier 
communication,  greater  trust,  and  stronger  support; 

•  Generate  a  concise,  agreed-to  Terms  of  Reference,  covering  goals,  scope,  products,  schedule, 
and  resources; 

•  Find  out  at  an  early  stage  in  the  project  what  the  products  of  the  study  are  to  be  used  for  by  the 
sponsor  and  stakeholder  organisations; 

•  Early  in  the  study,  agree  on  a  common  language  and  develop  a  study  glossary.  This  should  evolve 
during  the  course  of  the  study;  and 

•  Perform  a  rapid  first  pass  of  all  the  phases  of  the  project  to  help  establish  the  approximate  budget 
requirements,  especially  in  large  C2  projects.  The  assessment  team  should  establish  strategies  to 
address  shortfalls  in  the  event  the  sponsor’s  resources  are  limited  to  a  level  below  what  is  required 
to  support  a  quality  study. 

2.3  ASSESSMENT  TEAM 

The  precise  skills  and  experience  required  by  the  assessment  team  must  be  established  following  initial 
problem  formulation.  The  assessment  team  must  be  interdisciplinary. 
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As  an  illustration,  the  following  are  representative  skills  and  experience  needed  for  the  core  team  and 
consultants  and  part-time  team  members  for  a  recent,  complex  C2  assessment: 

•  Core  Team:  project  management;  OR/OA  skills;  cross  military  experience;  human  science  and 
organisational  theory;  data  collection;  and 

•  Consultants  and  Part-time  Team  Members:  military;  training  and  exercise  planning; 
communications  and  information  systems  expertise;  human  computer  interface  expertise;  OOTW 
related  issues. 

Sufficient  time  and  a  facilitating  process  should  be  built  into  the  project  plan  for  the  group  of  individuals 
to  coalesce  into  a  team. 


2.4  ASSESSMENT  PROCESS 

•  The  assessment  process  is  non-linear  and  iterative. 

•  The  assessment  team  must  realise  that  all  of  the  elements  of  the  C2  assessment  are  interrelated. 
Hence  Problem  Formulation,  Solution  Strategy,  Measures  of  Merit,  Scenarios,  Human/Organisational 
Factors,  Models  and  Tools,  Data,  and  products  are  all  interdependent  (See  Figure  1). 

•  Peer  review  is  a  necessity  not  a  luxury. 
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Chapter  3  -  PROBLEM  FORMULATION 


Effective  problem  formulation  is  fundamental  to  the  success  of  all  assessments,  but  particularly  in  C2 
assessment  because  the  issues  are  often  ill-defined  and  complex,  involving  many  dimensions  and  a  rich 
context.  The  assessment  team  must  perform  problem  formulation  carefully  and  understand  its  underlying 
principles.  There  are  principles  of  problem  formulation  that  apply  to  C2  assessment,  in  general,  some  of 
which  are  particularly  important  to  C2  assessment  for  OOTW,  in  particular.  Drawing  on  these  principles, 
one  can  characterise  the  problem  formulation  process  and  the  products  that  are  to  be  produced  in  the 
problem  formulation  phase. 

3.1  DEFINITION  OF  PROBLEM  FORMULATION 

•  Problem  formulation  is  an  iterative  process  that  evolves  over  the  course  of  the  study.  Iteration  is 
essential  even  for  small  studies  or  where  time  is  short  -  it  will  save  time  later  and  help  ensure  quality. 

•  Problem  Formulation  is  fundamentally  a  social  process  of  developing  a  shared  understanding. 

•  The  context  of  the  study  includes  geopolitical  context;  political,  social,  historical,  economic, 
geographic,  technological  environments;  actors;  threats;  aims  and  objectives  of  the  assessment, 
including  the  decisions  to  be  supported;  generic  C2  issues;  relevant  previous  studies;  and  stakeholders 
and  their  organisational  affdiations. 

•  The  aspects  of  the  problem  include  issues  to  be  addressed;  assumptions;  high-level  Measures  of  Merit 
(MoM);  independent  variables  (controllable  and  uncontrollable);  constraints  on  the  values  of 
the  variables  (domain  and  range);  time  constraints  on  delivery  of  advice  to  the  decisionmaker; 
and  whether  this  is  a  single  decision  or  (possibly  one  of)  a  chain  of  decisions  to  be  made  over  time. 

•  The  problem  is  not  formulated  until  the  assessment  team  has  addressed  each  aspect  of  the  problem. 

3.2  PRINCIPLES  OF  PROBLEM  FORMULATION 

•  Proper  problem  formulation  takes  substantial  time  and  effort! 

•  Explicit  problem  formulation  must  precede  construction  of  concepts  for  assessment  or  method 
selection. 

•  The  assessment  team  must  have  an  understanding  of  the  decisions  to  be  supported  by  the  assessment 
and  the  viewpoints  of  the  various  stakeholders  to  clarify  the  study  issues. 

•  Problem  formulation  must  not  only  provide  problem  segments  amenable  to  analysis,  but  also  a  clear 
and  valid  mechanism  for  meaningful  synthesis  to  provide  coherent  knowledge  about  the  original, 
larger  problem. 

•  Problem  formulation  must  be  broad  and  iterative  in  nature,  accepting  the  minimum  of  a  priori 
constraints  and  using  methods  to  encourage  creative  and  multi-disciplinary  thinking. 

•  The  problem  formulation  process  should  not  focus  prematurely  on  subsets  of  the  problem. 

•  Practical  constraints  such  as  data  availability,  study  resources  (including  time),  and  limitations  of  tools 
should  be  treated  as  modifiers  of  the  problem  formulation  rather  than  initial  drivers.  Such  constraints 
may,  in  the  end,  drive  the  feasible  solutions,  but  it  is  important  to  recognise  this  as  a  compromise 
rather  than  an  ideal. 
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•  Problem  formulation  should  address  risk  from  multiple  perspeetives.  Risk  analysis  techniques  should 
be  used  to  directly  explore  options  to  mitigate  risk. 


3.3  PRINCIPLES  PARTICULARLY  APPROPRIATE  FOR  OOTW  C2 
ASSESSMENTS 

•  Problem  formulation  must  address  the  geopolitical  context  of  the  OOTW  problem  and  seek  to  identify 
the  “broad”  C2  issues  contained  within  the  Terms  of  Reference  for  the  study. 

•  OOTW  C2  assessments  often  involve  policy-related  impacts  outside  the  context  of  a  particular 
military  operation.  Therefore,  MoM  hierarchies  must  contain  Measures  of  Policy  Effectiveness. 

•  The  assessment  team  must  have  an  historical  perspective  to  understand  OOTW  issues  because  social 
conflict  and  structures  often  have  roots  far  back  in  history. 

•  The  assessment  team  must  have  access  to  subject  matter  experts  from  a  broad  range  of  disciplines 
(e.g.  social  scientists,  historians,  and  regional  experts  in  OOTW  assessment). 

3.4  PROBLEM  FORMULATION  PROCESS 

The  process  begins  with  the  sponsor  presenting  the  assessment  team  with  a  problem  to  assess  and  an 
articulation  of  broad  constraints. 

During  the  early  stages  of  problem  formulation  the  assessment  team  should  quickly  cover  the  whole 
assessment  process  and  produce  an  initial  formulation. 

The  assessment  team  must  identify  what  it  perceives  as  the  real  issues  to  address.  The  team  must  engage  in 
a  dialogue  with  the  key  sponsor  and  stakeholders  to  get  “buy  in”  for  these  issues. 

In  dealing  with  fuzzy  or  uncertain  boundaries,  the  assessment  team  should  explore  and  understand  the 
significance  of  each  proposed  boundary.  The  assessment  team  should  keep  an  open  mind,  during  the  early 
stages  of  problem  formulation,  about  where  the  boundaries  lie  and  their  dimensional  nature. 

While  clear  definitions  and  hard  conceptual  boundaries  are  ultimately  necessary  in  order  to  create  a 
manageable  problem  space,  the  assessment  team  should  avoid  coming  to  closure  prematurely. 

Identification  of  high-level  MoM  should  start  with  ideal  measures  of  the  desired  benefits  before 
considering  what  can  be  practically  generated  by  analysis  (the  latter  may  force  the  use  of  surrogate  MoM, 
but  these  must  be  clearly  related  to  the  desired  measures). 

The  assessment  team  should  identify,  develop  (if  necessary),  and  apply  appropriate  tools  to  support 
problem  formulation.  Representative  tools  and  techniques  include:  techniques  for  supporting  expert 
elicitation,  influence  diagrams,  causal  maps,  system  dynamics  models,  and  agent-based  models. 


3.5  PROBLEM  FORMULATION  PRODUCTS 

An  iteration  of  the  problem  formulation  process  can  be  said  to  be  complete  when  the  following  is 
accomplished  and  documented:  the  “real”  question  to  be  answered  is  known;  the  assumptions  have  been 
articulated;  and  the  high  level  MoM,  the  independent  variables,  and  the  constraints  associated  with  the 
variables  have  been  identified. 
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Chapter  4  -  SOLUTION  STRATEGIES 


The  Problem  Formulation  phase  elarified  “what”  is  to  be  aehieved  during  the  assessment.  The  Solution 
Strategy  phase  must  transform  this  understanding  into  “how”  these  goals  and  objectives  are  to  be 
achieved.  Even  if  the  way  ahead  seems  clear,  the  articulation  of  a  formal  solution  strategy  is  necessary. 
Thus,  the  assessment  team  must  understand  the  definition  and  principles  of  the  solution  strategy, 
the  process  by  which  it  is  developed,  and  the  products  that  it  produces. 


4.1  DEFINITIONS  AND  PRINCIPLES 

A  solution  strategy  consists  of  the  specification  of  a  set  of  sequential  and  parallel  analytical  steps 
(documented  in  the  Study  Plan),  often  involving  several  methodologies  and  tools.  The  solution  strategy  is 
designed  to  begin  with  what  is  known,  and  by  execution  of  the  specified  steps,  leads  to  what  one  desires  to 
know  -  an  illumination  of  the  issues. 

•  The  Study  Plan  consists  of  two  inter-related  parts  -  the  formulated  problem  (the  What)  and  the 
solution  strategy  (the  Flow). 

•  The  solution  strategy  should  not  be  designed  before  an  initial  pass  through  the  problem  formulation 
process  and  the  problem  formulation  products  are  available  to  the  team. 

•  The  assessment  team  should  always  remember  the  inherently  iterative  nature  of  the  process. 

4.2  DEVELOPING  A  SOLUTION  STRATEGY 

•  The  solution  strategy  should  strike  an  artful  balance  between  what  the  team  would  like  to  do  and  what 
is  possible  to  do,  given  the  state  of  the  art,  available  data,  tools,  schedule,  and  resources  available. 

•  The  team  should  first  elaborate  on  the  measures  that  are  to  be  evaluated  in  the  study.  Using  these 
measures  and  consideration  of  human  and  organisational  factors,  a  conceptual  model  of  the  analysis 
should  be  developed  and  refined  as  data  requirements,  methods  and  tools,  and  scenarios  are  selected 
for  the  analysis. 

•  Frequently  a  solution  strategy  becomes  complex,  thereby  requiring  the  team  to  decompose  the 
problem  into  parts,  each  of  which  requires  assessment  with  its  own  set  of  tools. 

•  Taken  together,  the  solution  strategy  must  include  the  MoMs,  relevant  human  and  organisational 
factors,  specification  of  scenarios,  data  collection  requirements,  and  methods  and  tools  to  be  used  in 
the  analysis. 

4.3  PLANNING  DOCUMENTS 

The  solution  strategy  is  documented  in  a  Study  Plan  that  links  the  problem  formulation  and  solution 
strategy  together  in  one  plan. 

The  Study  Plan  should  be  developed  in  an  iterative  fashion,  applying  guidance  and  feedback  received 
from  the  study  sponsor  and  the  study  stakeholders. 
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The  Study  Plan  is  typically  approved  and  signed  by  the  sponsor  (decisionmaker).  Often  the  Study  Plan  is 
supported  by  a  Study  Management  Plan  to  guide,  manage,  and  coordinate  the  efforts  of  the  effort. 

The  Study  Management  Plan  may  have  subordinate  plans,  to  include  an  Analysis  Plan(s),  Modelling 
and  Simulation  Plan,  Data  Collection  Plan,  Configuration  Management  Plan,  Quality  Assurance  Plan, 
Review  Plan,  Deliverable  Plan,  Security  Plan,  a  Study  Risk  Register,  and  Glossary. 
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Chapter  5  -  MEASURES  OF  MERIT 


5.1  DEFINITIONS 

MoM  is  a  generic  term  to  encompass  different  classes  of  measures.  The  measures  are  defined  in 
hierarchical  levels  related  to  each  other,  each  in  terms  of  its  own  boundary.  An  orchestrated  set  of  MoMs 
is  typically  required  for  C2  assessments.  The  COBP  has  adopted  the  following  hierarchical  set  MoMs: 

•  Measures  of  Policy  Effectiveness  (MoPE)  that  focus  on  policy  or  societal  outcomes 
(e.g.  transition  measures,  which  focus  on  the  progress  in  the  transfer  of  responsibilities  to  a 
follow-on  military  force  or  civil  agency,  and  normality  indicators  which  measure  the  quality  of 
life  of  the  civilian  population); 

•  Measures  of  Force  Effectiveness  (MoFE)  that  focus  on  how  a  force  performs  its  mission  or  the 
degree  to  which  it  meets  its  objectives  (e.g.  loss  exchange  ratios,  combat  effectiveness,  number  of 
targets  destroyed  and  desirable  adversary  behaviour); 

•  Measures  of  C2  Effectiveness  (MoCE)  that  focus  on  the  impact  of  C2  systems  within  the 
operational  context  (e.g.  time  to  develop  a  Course  of  Action,  ability  to  provide  information  in 
required  format,  impact  of  information  operations,  and  planned  quality); 

•  Measures  of  Performance  (MoP)  that  focus  on  internal  system  structure,  characteristics, 
and  behaviour  (e.g.  time  to  recognise  an  event,  correctness  of  perception  and  system  reliability); 
and 

•  Dimensional  Parameters  (DP)  that  focus  on  the  properties  or  characteristics  inherent  in  the  C2 
system  (e.g.  bandwidth,  data  access  times,  cost,  and  size;  characteristics  of  organisation  forms, 
attributes  of  personnel). 


Environment 


5.2  PRINCIPLES 

•  A  multi-faceted  and  sometimes  multi-phased  approach  is  recommended  as  no  single  measure  or 
methodology  exists  to  satisfactorily  assess  the  overall  effectiveness  of  C2. 
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•  Established  objectives  for  the  assessment  must  directly  link  to  the  MoMs. 

•  Selection  of  MoMs  should  consider  assumptions,  constraints,  models,  tools,  scenarios,  other  elements 
of  the  analytic  plan,  and  assessment  processes. 

•  Identify  selected  MoMs,  their  thresholds  and  standards,  their  means  of  collection,  their  relationship  to 
the  assumptions,  and  their  imposed  constraints  in  the  assessment. 

•  A  detailed  review  of  reliability  and  validity  for  the  selected  measures  will  determine  a  level  of 
confidence  for  each  MoM. 

•  The  analyst  must  establish  and  measure  control  variables  to  correlate  MoMs  in  a  spectrum  of  multiple 
scenarios. 

•  The  analyst  must  pay  particular  attention  to  measurements  related  to  the  human  element, 
since  variations  in  measurements  may  well  cause  unacceptable  levels  of  uncertainty. 

•  Analysis  of  uncertainties  and  measures  of  central  tendency  and  dispersion  are  significant  for  C2 
issues. 

•  For  C2  acquisition  analyses,  measures  should  be  generated  in  parallel  with  system  development, 
so  they  can  be  used  as  standards  for  system  tests  and  operations. 
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Chapter  6  -  HUMAN  AND  ORGANISATIONAL  FACTORS 


The  human  dimension  largely  distinguishes  C2  analysis  from  other  military  operations  analysis. 
C2  analysis  must  deal  with  distributed  teams  ineluding  military,  interagency,  coalition  and  other  non-state 
actors  operating  under  stress  and  their  varying  decisionmaking  behaviours.  In  OOTW,  particular  attention 
must  be  paid  to  behaviour  of  and  interaction  with  non-military  organisations,  political  groups, 
and  amorphous  groups  such  as  crowds  and  refugees.  Thus,  the  formulation  of  the  problem  and  the 
development  of  solution  strategies  cannot  be  completed  without  explicit  consideration  of  both  human  and 
organisation  issues. 

•  Human  and  organisational  factors  must  be  considered  as  part  of  structuring  the  problem,  selecting 
MoMs,  defining  scenarios,  developing  solution  strategies,  and  selecting  methods  and  tools,  and  they 
should  be  reviewed  throughout  the  entire  analytical  process. 

•  The  assessment  team  must  include,  or  have  access  to,  experts  from  organisational  science  and  the 
various  human  science  disciplines  (such  as  cultural  anthropology,  demography,  sociology,  social  and 
individual  psychology,  political  science). 


6.1  HUMAN  FACTORS 

Human  factors  of  interest  fall  into  the  major  categories: 

•  Human  behaviour  related  to  performance  degradation  and  as  a  consequence  of  social  interaction 
among  individuals  or  members  of  a  group; 

•  Decisionmaking  behaviour  (cognitive  questions)  including  the  cognitive  complexity  of  issues  and 
the  capacities  of  the  commanders  and  other  decisionmakers  of  interest;  and 

•  Command  style. 

Human  performance  depends  on  psycho-physiological  variables  (e.g.  stress,  sleep  deprivation,  hunger, 
and  alertness)  and  on  ergonomic  and  external  factors.  Individual  and  group  behaviour  is  the  result  of  social 
processes  and  factors  (e.g.  fear,  morale,  values),  and  the  cultural,  educational,  and  religious  background  of 
individuals. 

•  Any  time  human  performance  and/or  behaviour  are  at  issue,  parameters  and/or  models  are  needed  to 
reflect  those  issues.  Unless  specialists  can  provide  valid  parameters  from  work  in  other  contexts  or 
from  field  experience  in  OOTW,  some  experimentation  may  be  necessary  and  appropriate  to  develop 
them. 

•  Decisionmaking  that  is  automatable,  contingent,  rule  or  algorithmically  based  can  be  modelled 
directly,  but  time  requirements  and  error  representation  must  be  incorporated  if  humans  are  involved 
in  the  actual  process. 

•  Complex  decisions  (e.g.  courses  of  action  in  response  to  events  in  the  operations  space)  are  best  being 
modelled  with  “human  in  the  loop”  techniques.  Closed  form  techniques  for  modelling  complex 
decisions  are  still  in  the  experimental  stage. 

•  Accounting  for  differences  in  command  styles  and  how  they  affect  military  decisionmaking  is  of 
special  importance  in  OOTW  which  tend  to  be  multinational  coalition  operations.  They  may  be 
reflected  by  attributes  such  as  the  background  of  commanders,  their  field  experience,  risk  attitudes, 
and  organisational  and  orders  style. 
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6.2  ORGANISATIONAL  FACTORS 

Organisational  design  reflects  the  interaction  among  the  tasks  to  be  done,  the  people  to  do  them,  and  the 
systems  and  tools  that  support  those  people.  The  principal  differences  between  organisations  are  related  to 
structure,  function  and  capacity. 

•  Structural  differences  include: 

•  Number  of  echelons  in  the  command  structure; 

•  The  span  of  control  for  nodes  in  the  command  structure;  and 

•  The  pattern  of  (formal  and  informal,  permanent  and  transitory)  linkages  between  nodes 
(hierarchical  tree,  spokes  of  a  wheel,  multi-connected,  networked). 

•  Functional  differences  include: 

•  The  distribution  of  responsibility  (location  of  functional  activities  such  as,  e.g.,  intelligence, 
logistics,  civil  military  cooperation  (CIMIC); 

•  The  distribution  of  authority  (ideally  co-located  with  responsibility); 

•  Functional  specificity  (of  combat,  support,  or  service  capabilities)  versus  integrated  capabilities 
(mission  tailored  task  forces);  and 

•  Degree  of  ambiguity  in  command  relationships. 

•  Capacity  differences  are  related  to: 

•  Personnel  (quality,  training,  experience); 

•  Information  and  communications  systems  and  architectures;  and 

•  C2  field  training  and  operational  experience. 

A  systematic  approach  using  a  hypothesis-testing  logic  and  aided  by  organisation  theory  expertise  should 
be  used  for  addressing  organisational  issues  in  C2  assessment  because  of  the  large  number  and  many 
indirect  effects  of  organisational  variables. 
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Chapter  7  -  SCENARIOS 


Scenarios  provide  the  context  for  the  conduct  of  the  operational  analysis  and  bound  the  arena  of 
the  analysis.  Scenarios  consist  of  several  static  dimensions  shown  in  the  table  and  include  the  dynamic 
evolution  of  events  in  time.  Operational  scenarios  detail  threats,  orders  of  battle,  tactics,  rules  of 
engagement  and  courses  of  action,  deployments,  reserves,  adversary  forces,  and  non-combatants. 


Table  1:  The  Static  Dimensions  of  the  Scenario  Framework 


External 

Factors 

Environment 

Mission  Objectives 

Mission 

Economic/Military/ 

Political/Social/ 

Historic  Situation 

Mission  Constraints  & 
Limitations  ROE 

Military  Scope 

Intensity  }  Tasks 

Joint/Combined  ’ 

National  Security  Interests 

Capabilities 
of  Actors 

•  Organisation,  Order  of  Battle,  02,  Doctrine,  Resources,  Lessons  Learned 

•  Weapons 

•  Logistics 

•  Skills 

•  Morale 

Friendly  Forces  Adversary  Forces  Neutral  Forces  Non-Combatants 

Environment 

•  Geography/Region/Terrain 

•  Climate/Weather 

•  (Civil)  Infrastructure  (e.g..  Transportation,  Telecommunications,  Energy) 

7.1  ATTRIBUTES 

Several  prerequisites  are  essential  before  using  scenarios  for  C2  analysis: 

•  Approval:  the  analyst  should  strive  for  the  creation  of  a  family  of  scenarios  consistent  with 
high-level  guidance  and  policy; 

•  Breadth:  scenarios  should  reflect  those  factors  that  are  hypothesised  to  have  a  significant  impact 
on  C2  issues; 

•  Capability:  scenarios  should  stress  C2  capabilities,  including  human  and  organisational  factors 
(military  and/or  civilian)  where  appropriate; 

•  Credibility:  scenarios  should  include  logical  assumptions  about  the  problem  under  analysis;  and 

•  Plausibility:  scenarios  should  represent  plausible  situations. 


7.2  PRINCIPLES 

•  Analysts  need  to  use  multiple  scenarios  and  vignettes  to  cover  or  sample  the  interesting  problem  space 
since  no  single  scenario  is  sufficient. 

•  Analysts  should  explicitly  identify  and  describe  the  scenarios  prior  to  the  execution  of  a  study.  It  is 
best  practice  to  revisit  the  scenario  definition  periodically  during  the  conduct  of  the  study. 
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•  For  coalition  C2  assessments,  scenarios  should  be  developed  or  adapted  by  teams  with  representatives 
from  all  participating  nations. 

•  Scenarios  should  reflect  C2  organisations  and  infrastructure  (including  human  issues),  processes, 
and  systems  relevant  to  the  analysis. 

•  Scenarios  must  consider  mission  scope,  levels  of  hierarchy,  and  data  flow  aggregation/ 
disaggregation. 

•  Key  scenario  assumptions  and  constraints  should  be  identified  and  documented. 

•  Boundaries  of  the  scenario  spaces  should  be  defined  by  the  problem  being  analysed. 

•  Scenarios  should  reflect  the  factors  that  have  significant  impact  on  C2,  stress  C2  issues,  are  credible  to 
the  military,  and  are  credible  in  terms  of  civil-military  objectives. 
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Chapter  8  -  METHODS  AND  TOOLS 


This  section  considers  the  best  methods  and  tools  (quantitative  or  qualitative)  for  assessing  C2  processes, 
performance  and  effectiveness.  The  section  covers  methods  and  tools  used  for  analysis,  training  or 
operations,  each  of  which  has  different  requirements.  Available  methods  can  be  categorised  into  several 
classes,  namely:  data  collection/generation,  data  organisation/relationship,  solving  and  support.  Table  2 
illustrates  this  categorisation  for  a  sample  of  methods. 


Table  2:  Examples  of  Methods  and  Tools  Categorised  by  Use 


Data 

Generation 

Data 

Collection 

Data 

Organisation/ 

Relationship 

“Solving” 

Support 

After  Action  Reviews  and 
Historicai  Anaiysis 

• 

• 

Expert  Eiicitation 

• 

Constructive  and  Virtuai 
Simuiations 

• 

• 

• 

Exercises/Experiments 

• 

• 

Game  Theory 

• 

• 

Army  C2  Evaiuation 
System  (ACCES) 

• 

• 

Causai  Mapping 

• 

Muiti-Criteria  Decision 
Anaiysis  (MCDA) 

• 

• 

Regression  Anaiysis 

• 

• 

Bayesian  Networks 

• 

• 

Mathematicai 

Programming 

• 

• 

Heuristic  Search 

• 

Genetic  Aigorithms 

• 

• 

Project  Management 

Toois 

• 

Data  Anaiysis 

• 

Geographicai 
information  Systems 

• 

Visuaiisation 

• 

• 

Databases 

• 

• 

Checkiists 

• 

A  key  challenge  for  C2  assessment  methods  is  to  make  properly  quantified  linkages  between  MoP,  MoCE, 
MoFE,  and  MoPE.  All  C2  assessments  require  a  high  level  of  creative  problem  structuring  and  solving  in 
making  these  linkages. 

8.1  METHOD  AND  TOOL  SELECTION 

The  following  issues  must  be  considered  when  selecting  methods  and  tools  for  C2  assessments: 

•  The  inherent  complexity  of  C2  assessment  problems,  which  calls  for  an  orchestrated  set  of 
complementary  tools  to  cover  the  wide  range  of  variables  involved  (including  hierarchies  or 
federations  of  models  rather  than  dealing  with  all  issues  in  a  single  model); 
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•  The  requirement  to  explore  a  wide  range  of  scenarios  and  still  represent  C2  processes  and 
performance  effectively; 

•  The  appropriate  treatment  of  human  issues  in  models  and  the  balance  of  realism  against  possible 
sources  of  uncertainty  in  the  use  of  “human-in-the-loop”  techniques; 

•  The  relative  advantages  of  stochastic  and  deterministic  models  (depending  on  the  nature  of  the 
assessment  problem  and  other  elements  of  the  solution  approach); 

•  The  appropriate  balance  for  the  representation  of  friendly,  adversary  and  other  forces;  and 

•  The  challenge  of  VV&A  and  the  development  of  trust  in  methods  and  tools,  especially  in  novel 
applications  areas  such  as  unfamiliar  OOTW  situations. 

Best  practice  for  the  application  of  analysis  tools  for  C2  assessment  is  still  emerging,  but  the  following 
criteria  should  be  used  when  selecting  models: 

•  Functionality-based  criteria  (including  resolution,  completeness/scope,  functionality,  explicitness, 
the  ability  to  generate  required  MoM,  validity  and  accreditation);  and 

•  Performance-related  criteria  (including  responsiveness,  simplicity,  preparation/use  time,  data 
availability,  interoperability  with  other  tools,  resource  requirements  and  credibility). 

8.2  UTILITY  OF  DIFFERENT  TYPES  OF  METHODS  AND  TOOLS 

•  The  use  of  model  federations,  particularly  with  an  object-oriented  approach,  encourages  development 
using  holistic  and  evolutionary  principles,  under  which  modellers  should  capture  a  complete  model  of 
the  process,  including  parts  whose  representation  is  still  unclear  (ready  for  improvement  as 
understanding  develops). 

•  Agent-oriented  modelling  favours  the  capture  of  the  cognitive  nature  of  command  tasks.  This  is 
important  in  establishing  the  linkage  between  MoP  and  MoFE,  but  is  constrained  by  the  difficulty  in 
representing  human  performance,  command  styles  and  organisational  relationships. 

•  Very  fast-running  models  can  be  used  as  screening  tools  during  problem  formulation  to  scan  the 
problem  space,  allowing  identification  of  areas  of  concern  for  deeper  analysis. 

•  The  model-test-model  or  model-exercise-model  processes  should  be  considered  where  the  assessment 
problem  requires  generation  of  new  data  to  validate  existing  models  of  C2  factors.  Assessment  teams 
should  be  aware  of  the  latest  advances  in  fast,  agile  modelling  and  exploit  them. 

8.3  DIRECTIONS  FOR  FUTURE  DEVELOPMENT 

The  following  ideas  should  be  considered  in  new  model  developments  (taking  account  of  the  appropriate 
level  of  model  aggregation): 

•  Understanding  of  adversary  intent  can  be  represented  by  having  a  set  of  prescribed  options; 

•  FlQs  with  a  local  “picture”  in  each  should  be  explicitly  represented  to  allow  different  perception- 
based  behaviours  to  be  represented,  facilitating  the  study  of  factors  such  as  deception,  shock, 
and  surprise;  and 

•  Information  should  be  represented  as  a  commodity  as  follows: 

•  A  realistic  flow  of  information  around  the  operational  environment; 

•  The  collection  of  information  from  multiple  sources  and  the  tasking  of  information  collection 
assets; 
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•  The  processing  of  information; 

•  C2  systems  as  entities  on  the  battlespace/operational  space; 

•  Unit  perceptions  built,  updated,  and  validated  from  the  information  available  to  the  unit 
through  its  information  systems; 

•  The  commander’s  decision  based  on  the  unit’s  perception  of  the  operational  space;  and 

•  Information  operations  actions  and  impacts  across  all  sides  represented  in  the  model. 
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Chapter  9  -  DATA,  METADATA,  AND  THE 
COMMON  DATA  INFRASTRUCTURE 


While  the  value  of  data  for  an  individual  study  effort  is  well  understood  by  the  analytic  community  at 
large,  the  aggregated  worth  of  data  is  still  undervalued  by  many.  Data  can  be  described  as  the  fundamental 
elements  of  information  and  knowledge  that  comprises  the  corporate  whole  -  consequently  its  aggregated 
value  particularly  when  addressed  in  a  context  larger  than  an  individual  study  is  significantly  greater  than 
the  sum  of  the  parts.  Explicitly  dealing  with  this  issue  resulted  in  data  receiving  its  own  chapter  in  the 
code. 

Data  are  factual  information  organised  for  analysis. 

•  The  Data  Taxonomy  in  the  COBP  lays  out  a  number  of  types  of  data  including  broad  categories  of 
sources  that  will  be  of  interest  to  the  analyst.  The  ability  to  determine  the  needed  data  and  the  ability 
to  assemble  or  collect  these  data  will  constrain  the  solution  strategy. 

•  A  major  challenge  is  to  find,  organise,  verify,  process,  and  convert  source  data  into  the  data  needed 
for  the  study.  The  team  needs  to  know  (a)  what  data  are  needed  in  which  structure; 
(b)  who  owns  these  data;  (c)  security  issues,  (d)  costs  to  buy,  collect,  or  generate  data. 

•  If  the  data  are  not  available  and  can  neither  be  aggregated  nor  derived  from  the  available  sources,  it  is 
good  practice  to  use  the  knowledge  of  subject  matter  experts  to  generate  the  necessary  data. 

Meta-data  are  “information  about  information,”  which  are  used  to  document  the  data  and  related  issues. 

•  The  source  of  the  data,  the  reliability,  and  assorted  assumptions  and  constraints  must  be  captured  in 
standardised  meta-data  sets  assigned  to  the  data  sets  comprising  the  needed  information. 

•  Data  that  are  collected  without  adequate  documentation  are  frequently  viewed  as  suspect  or  unusable. 
To  avoid  having  good  data  thrown  away  due  to  the  lack  of  documentation,  acceptable  community 
standards  for  documentation  must  be  employed.  This  information  has  to  be  captured  in  standardised 
meta-data  sets. 

•  The  initial  data  available  will  often  be  vague,  uncertain,  incomplete,  and  contradictory.  On  the  other 
hand  analysts  prefer  data  to  be  sharp,  certain,  complete,  and  consistent.  The  assumptions  inherent  in 
accomplishing  this  transformation  must  be  captured  in  standardised  meta-data  sets. 

A  Common  Data  Infrastructure  facilitates  reuse  of  data.  It  is  based  on  agreed  standards  for  data  formats, 
meta-data,  and  documentation. 

•  Data  are  central  to  the  conduct  of  good  assessments.  Adhering  to  data  engineering  principles 
contributes  to  data  reuse.  Archiving  of  data  in  retrievable  form  using  standardised  meta-data  sets  is 
one  essential  component  to  facilitate  data  reuse. 

•  As  the  data  being  used  today  by  the  analysts  will  be  the  data  needed  tomorrow  by  systems  engineers, 
decisionmakers,  and  commanders  for  their  operations,  it  is  good  practice  to  align  the  standardisation 
processes  and  respective  tool  sets  between  the  analytical  and  operational  communities  in  order  to 
facilitate  the  transfer  of  data  between  those  communities. 
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Chapter  10  -  RISK  AND  UNCERTAINTY 

10.1  DEFINITION 

Risk  is  commonly  defined  as  the  possibility  of  suffering  harm  or  loss.  Uncertainty  can  be  defined  as  an 
inability  to  determine  a  variable  or  system  state  or  predict  its  future  evolution. 

There  are  risks  associated  with  the  uncertainties  in  decisionmaker’s  situation  that  are  the  subject  of  the 
assessment  and  there  are  risks  related  to  the  conduct  of  the  assessment  itself  Using  the  COBP  should  help 
minimise  the  risks  involved  in  C2  studies,  but  the  following  risk  and  uncertainty  issues  remain  and  are 
intrinsic  to  any  C2  study. 


10.2  SCOPE  OF  RISK  AND  UNCERTAINTY 

•  The  explicit  treatment  of  risk  and  uncertainty  is  best  practice  in  all  studies  and  of  particular 
importance  in  C2  assessment.  It  is  advisable  not  to  skip  risk  analysis,  even  when  time  and  resources 
are  limited,  and  this  should  be  stressed  to  decisionmakers  and  study  sponsors. 

•  Perceptions  of  risk  and  uncertainty  can  substantially  differ  from  objective  assessments  and  this  should 
be  explicitly  considered.  Analysts  should  find  out  how  study  sponsors  perceive  risks. 

•  A  thorough  understanding  of  study  variables  is  essential  for  effective  treatment  of  uncertainty. 
OOTW  studies  typically  have  less  well-formed  quantitative  factors  and  more  qualitative  factors. 

•  The  assessment  team  must  be  aware  of  sources  of  uncertainty  in  all  aspects  of  a  study,  including 
those  related  to  the  assumptions  and  limitations  inherent  in  parameter  values,  models,  scenarios, 
data  structures,  and  the  boundaries  used  to  scope  and  focus  the  study.  Humans  involved  in  the 
assessment  process  make  assumptions  that  should  be  identified,  documented  and  analysed. 

•  In  C2  assessments,  analysts  need  to  be  particularly  alert  to  the  possibility  of  chaotic  behaviours  arising 
from  dynamic  interactions  of  human  and  organisational  factors,  and  discontinuous,  non-linear 
divergence  arising  from  the  multiple  options  for  choice  of  course  of  action. 

10.3  TREATMENT  OF  RISK  AND  UNCERTAINTY 

•  Where  uncertainties  cannot  be  reduced  by  acquiring  more  information,  multi-factorial  sensitivity 
analysis  should  be  used  to  establish  the  regions  for  which  the  results  are  valid  and  to  isolate  those 
factors  that  may  introduce  uncertainty. 

•  Sensitivity  analysis  should  not  only  deal  with  statistical  variance  but  should  also  consider  qualitative 
variations  in  models,  perspectives  and  assumptions.  A  range  of  analytic  tools  relevant  to  sensitivity 
analysis  is  identified  in  the  full  text  of  the  COBP. 

•  Checklists  and  structured  appraisal  help  to  maintain  study  rigour,  but  neither  is  a  substitute  for  critical 
thinking  in  the  specific  study  context. 

•  The  team  should  expect  a  complex  and  partly  hidden  set  of  risks  to  C2  studies  and  serious  efforts 
should  be  made  to  illuminate  them  in  an  explicit  risk-based  analysis  with  portfolio-based  solutions. 
Such  analysis  needs  metrics  for  risk  and  failure  as  well  as  MoMs. 


RTO-TR-081 


ASG  - 19 


NATO  CODE  OF  BEST  PRACTICE 

FOR  C2  ASSESSMENT  -  ANALYST’S  SUMMARY  GUIDE 


ORGAmZATION 


•  The  assessment  team  should  make  a  eomplete  list  of  risks  and  treat  them  in  appropriate  detail  in  a  risk 
register  for  the  study.  A  generie  risk  register  has  been  produced  to  support  the  COBP. 


10.4  COMMUNICATION  OF  RISK  AND  UNCERTAINTY 

•  Communication  of  uncertainty  and  risk  is  particularly  important  for  C2  assessment  products. 
Such  communication  must  take  account  of  the  complexities  of  the  subject  and  human  limitations  in 
understanding  risk  and  uncertainty. 

•  A  continuing  dialogue  with  stakeholders  about  uncertainty  will  facilitate  common  understanding, 
including  the  impact  of  uncertainty  on  the  robustness  of  conclusions  and  methods  for  mitigating  them. 


ASG  -  20 


RTO-TR-081 


NATO  CODE  OF  BEST  PRACTICE 
FOR  C2  ASSESSMENT  -  ANALYST’S  SUMMARY  GUIDE 


Chapter  11  -  PRODUCTS 


Assessment  produets  eommunieate  results  to  deeisionmakers  and  stakeholders,  establish  the  eredibility  of 
the  effort,  and  provide  a  lasting  reeord  of  the  project.  Typical  C2  assessment  products  include  the  Study 
Plan,  Periodic  Status  Reports,  the  Project  Journal  and  a  Final  Report  and  Briefing,  which  include  a  variety 
of  supporting  data  and  documents: 

•  The  Study  Plan,  a  living  document  kept  current  throughout  the  assessment,  explains  the  problem 
under  analysis,  the  solution  strategy,  the  tasks  involved,  and  how  they  fit  together.  This  will 
include,  as  appropriate,  plans  necessary  for  the  assessment,  such  as  data  collection  and  data 
analysis  plans.  The  elements  of  a  Study  Plan  are  enumerated  in  the  Solution  Strategy  section; 

•  Periodic  Status  Reports  describe  the  state  of  the  effort  over  time  and  can  be  assembled  to  create  a 
record  of  the  work  accomplished,  problems  encountered,  results  from  peer  reviews,  and  the 
adjustments  made  by  the  team; 

•  The  Project  Journal  records  interactions  with  professionals  outside  the  core  team  and  captures  the 
analytic  assumptions  and  decisions  made  by  the  assessment  team  (such  as  choice  of  tools  and 
selection  of  data  or  methods  for  generating  data)  over  time; 

•  The  Final  Report  presents  the  results  of  the  effort  (findings,  recommendations,  and  lessons 
learned)  and  incorporates,  as  necessary,  items  from  the  other  key  documents.  It  will  also  typically 
include  appendices  that  provide  supporting  detail  such  as  participants  in  the  effort,  references, 
glossary,  list  of  acronyms  and  abbreviations,  as  well  as  technical  material  such  as  data  collection 
instruments; 

•  The  Final  Briefing,  which  typically  is  circulated  more  broadly  than  the  Final  Report,  will  provide 
a  summary  of  the  most  important  aspects  of  the  project,  including  the  problem  statement,  solution 
strategy,  research  accomplished,  findings,  recommendations,  and  lessons  learned.  Best  practice 
requires  that  this  document  be  richly  annotated;  and 

•  Data,  models,  supporting  scenario  materials,  and  other  products  necessary  to  make  the  results 
credible  and  authoritative  or  that  may  be  of  value  to  other  researchers  or  assessment  teams  should 
be  archived  and  made  available  as  broadly  as  possible. 

In  general,  all  the  products  should  be  circulated  as  widely  as  possible  so  that  others  can  benefit  from  the 
work  accomplished  and  redundant  projects  avoided.  This  requires  that  specific  effort  be  made  to 
downgrade  or  sanitise  products  and  to  identify  those  who  would  benefit  from  reviewing  them. 
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Annex  A  -  DECISIONMAKER’S  TEMPORAL  QUESTION  LIST 

Attached  is  a  list  of  questions  that  have  been  included  in  the  decisionmakers  ’  guide.  It  would  be  prudent  if 
the  analyst  is  prepared  to  respond  to  these  questions. 

This  annex  contains  key  questions  that  a  decisionmaker  should  ask  the  C2  assessment  team. 
These  questions  are  organised  temporally  according  to  the  following  phases  of  a  study  (prior  to  the  study, 
at  initial  review,  after  first  iteration,  and  at  final  report). 

Prior  to  the  Study: 

•  Do  you  understand  what  decisions(s)  I  have  to  make,  when  I  have  to  make  them,  and  the  context 
within  which  the  decision(s)  will  be  made? 

•  Do  you  need  any  information  or  authorisation  from  me? 

•  Who  will  be  on  the  study  team? 

•  Are  there  adequate  skills,  experience  present  in  the  team? 

•  For  OOTW  studies,  in  particular,  are  there  adequate  social  scientific  skills  in  the  team? 

•  Who  are  the  key  organisations/individuals  with  whom  you  plan  to  interact  (e.g.,  stakeholders, 
data  providers,  review  team)? 

•  Flave  you  coordinated  the  Terms  of  Reference  with  them? 

•  Flow  do  you  plan  to  interact  with  them? 

•  Particularly  for  OOTW  studies,  how  will  you  acquire  the  requisite  knowledge  of  the 
culture/historical  context? 

•  Flow  will  you  undertake  problem  formulation?  e.g., 

•  What  products  will  you  review/mine? 

•  What  methods  and  tools  are  applicable? 

•  When  will  key  events  occur?  (e.g.,  reviews,  production  of  interim  products) 

At  the  Initial  Review: 

•  What  do  you  perceive  the  “real”  issues  to  be? 

•  What  assumptions  do  you  plan  to  make  to  scope  the  effort? 

•  What  do  you  plan  to  use  for 

•  Fligh  level  MoMs? 

•  Scenarios  of  interest? 

•  Flave  you  identified  any  additional  organisations/individuals  with  whom  you  plan  to  coordinate? 

•  Flow  do  you  plan  to  attack  the  problem? 

•  What  methodology  will  you  employ? 

•  Flow  will  you  treat  the  diverse  aspects  of  a  mission  capability  package  (including  concept  of 
operations,  organisation,  doctrine,  C2  approach,  systems,  personnel,  facilities,  in  other  words 
everything  needed  to  field  a  real  capability)? 

•  Flow  do  you  plan  to  address  organisation/human  issues? 

•  What  specific  methods  and  tools  will  you  employ?  Why  do  you  think  they  are  appropriate? 
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•  What  data  will  you  employ?  Where  will  you  get  them?  Why  do  you  think  they  are  appropriate? 
How  do  you  plan  to  make  the  data  aeeessible  to  others?  How  do  you  plan  to  depiet  the  results  of 
the  study? 

At  First  Iteration: 

•  What  speeific  MoMs  were  seleeted?  What  relationships  were  established  among  the  MoMs? 

•  What  range  of  scenarios  were  selected?  Why? 

•  What  plans  do  you  have  to  illuminate  uncertainty/sensitivity? 

•  What  feedback  did  you  receive  from  the  independent  review  team?  What  steps  did  you  take  to 
respond  to  it? 

•  What  do  you  plan  to  do  on  subsequent  iterations? 

•  Use  additional  tools? 

•  Consider  additional  scenarios,  assumptions? 

•  Modify  assessment  boundaries? 


At  Final  Report: 

•  What  are  the  major  findings,  recommendations? 

•  What  are  the  key  points  of  uncertainty/sensitivity? 

•  What  issues  were  not  addressed  that  should  be  treated  in  subsequent  assessments? 

•  What  key  lessons  did  you  learn  with  respect  to  methods,  tools,  and  data? 

•  What  steps  are  you  going  to  take  to  disseminate  key  insights,  products  to  the  broader  community? 

•  Are  there  any  voids/issues  that  warrant  further  research? 
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Chapter  1  -  INTRODUCTION 


1.1  PURPOSE 

NATO  has  produced  a  Code  of  Best  Practice  (COBP)'  in  order  to  facilitate  high  quality  assessments 
supporting  decisionmaking  in  the  area  of  Command  and  Control  (C2).  The  COBP  is  the  product  of 
international  collaboration  drawing  together  the  operational  and  analytical  experience  of  leading  military 
and  civilian  defence  experts  from  across  the  NATO  nations.  The  COBP  enhances  the  understanding  of 
best  practice  and  outlines  a  structured  process  for  the  conduct  of  operational  assessment  for  C2,  which  is 
the  core  capability  of  Information  Age  defence  and  security. 

The  command  and  control  aspects  of  military  capability  are  difficult  to  assess.  Use  of  the  COBP  will: 

•  Increase  the  likelihood  of  quality  products; 

•  Complete; 

•  Relevant; 

•  Transparent  (visible); 

•  Credible  (believable); 

•  Authoritative  (rigorous);  and 

•  Reduce  risk/cost  associated  with  producing  the  products. 

The  COBP  emphasizes  the  critical  role  of  decisionmakers  in  the  conduct  of  good  assessment  and 
describes  a  framework  to  help  to  structure  this  participation.  The  COBP  should  be  used  by 
decisionmakers  to  frame  study  requirements,  provide  additional  guidance,  and  review  products.  Use  of  the 
COBP  should  be  regarded  as  a  community  standard  for  all  C2  assessments. 

This  decisionmaker’s  guide  aims  to  help  decisionmakers  who  commission,  fund,  oversee,  and  employ  C2 
assessments.  It  provides:  1)  an  executive  summary  of  the  NATO  Code  of  Best  Practice  (COBP)  for  C2 
Assessment,  2)  information  on  how  decisionmakers  can  best  ensure  that  the  COBP  is  adhered  to  by  those 
carrying  out  C2  assessments,  and  3)  guidance  on  the  limits  of  assessment  and  the  use  of  C2  assessment 
results. 


1.2  BACKGROUND 

The  initial  version  of  the  NATO  COBP  for  C2  Assessment  was  published  in  1998,  as  the  culmination  of 
several  efforts  by  various  NATO  study  groups  to  address  C2  assessment  challenges  and  practices. 
While  this  initial  version  focused  on  Article  V  related  operations,  this  version  of  the  COBP  addresses  all 
issues  of  C2  assessment  across  the  full  spectrum  of  operations,  to  include  Operations  Other  Than  War 
(OOTW).  Experience  with  the  initial  version  of  the  COBP  highlighted  the  need  to  produce  a  short, 
executive-level  version  of  the  COBP  aimed  specifically  at  decisionmakers. 


'  North  Atlantic  Treaty  Organisation  (NATO)  NATO  Code  of  Best  Practice  for  C2  Assessment,  2002. 
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1.3  COMMAND  AND  CONTROL  ASSESSMENT 

C2^  is  recognised  as  a  critical  element  of  successful  military  operations  and  a  key  aspect  of  Information 
Age  transformation.  Until  recently,  however,  physics-dominated  issues  of  military  operations,  rather 
than  C2  ones,  have  been  the  primary,  almost  exclusive,  focus  of  military  analysis  and  assessments. 
This,  coupled  with  the  inherent  complexity  of  C2  (which  involves  both  the  information  and  cognitive 
domains),  has  presented  the  assessment  community  with  challenges  that  are  less  well  researched  and 
understood  and  with  a  tool  kit  that  is  clearly  lacking.  The  NATO  COBP  for  C2  Assessment,  therefore, 
has  been  developed  to  help  C2  analysts  deal  with  these  new  Information  Age  assessment  challenges  so 
that  they  can  improve  their  ability  to  take  on  analyses  of  requirements,  analyses  of  alternatives,  research 
on  new  C2  concepts  and  capabilities,  and  support  real  world  operations.  This  guide  addresses  how  the 
decisionmaker,  as  a  customer  of  C2  assessments,  plays  a  key  role  in  such  assessments. 


1.4  ROLE  OF  THE  DECISIONMAKER 

The  decisionmaker  has  a  critical  role  in  any  assessment.  The  decisionmaker  provides  the  initial 
articulation  of  the  problem  or  issue  at  hand  and  establishes  the  conditions  under  which  the  effort  takes 
place.  In  addition,  it  is  the  decisionmaker  who  determines  how  the  assessment  results  are  interpreted  and 
whether  they  influence  decisions.  When  the  decisionmaker  interposes  subordinates  between  him  and  the 
assessment  team  he  must  understand  the  resulting  risks  and  take  action  to  mitigate  them  where  possible. 

The  decisionmaker’s  role  in  a  C2  assessment  is  most  definitely  a  hands-on  one.  Experience  shows  that  C2 
assessments  that  have  had  active  decisionmaker  participation  are  more  likely  to  result  in  products  that 
satisfy  both  decisionmakers  and  the  members  of  the  assessment  team.  Annex  A  provides  some  key 
questions  to  encourage  the  necessary  discussions  between  the  decisionmaker  and  the  assessment  team  at 
various  points  in  the  study  effort.  This  guide  will  provide,  at  a  high  level,  information  that  will  assist  the 
decisionmaker  in  assessing  the  answers.  The  full  COBP  provides  a  more  detailed  treatment  of  these  items 
and  the  decisionmaker  may  want  to  consult  the  full  COBP  as  appropriate. 

The  interaction  between  the  decisionmaker  and  the  assessment  team  is  not  only  critical  to  getting  the  effort 
off  on  the  right  foot  but  is  essential  if  the  decisionmaker  is  to  fully  understand  the  results  of  the  assessment 
and  the  assumptions  that  underlie  these  results.  Furthermore  the  assessment  team  will,  at  various  times 
during  the  study,  have  important  choices  to  make.  Decisionmaker  involvement  in  these  choices  can  make  the 
difference  in  a  study’s  success  or  failure. 


Establish  a  reasonable  set  of  initial  conditions 
Stay  involved  throughout  the  study 
Maintain  consistent  focus  on  the  real  problem 


1.5  ORGANISATION  OF  THE  GUIDE 

The  remainder  of  this  guide  is  organised  into  the  following  four  sections: 

•  Preparing  for  Success; 

•  Overview  of  the  Assessment  Process; 

•  Monitoring  the  Assessment;  and 

•  Reviewing  the  Products. 

2 

NATO  defines  C2  as  “The  Organisation,  Process,  Procedures,  and  Systems  necessary  to  allow  timely  political  and  military 
decisionmaking  and  to  enable  military  commanders  to  direct  and  control  military  forces.” 
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In  Preparing  for  Success,  the  things  that  the  decisionmaker  can  do  to  enhance  the  likelihood  that 
the  effort  is  successful  are  discussed.  In  the  Overview  of  the  Assessment  Process,  each  of  the  component 
steps  in  the  assessment  and  their  relationships  one  to  another  are  discussed.  In  Monitoring  the 
Assessment,  key  considerations  are  identified,  any  one  of  which  could  be  a  determining  factor  in  the 
success  or  failure  of  the  effort.  Reviewing  the  Products  addresses  the  nature  of  the  products  a  C2 
assessment  should  be  expected  to  produce. 
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Chapter  2  -  PREPARING  FOR  SUCCESS 


There  are  two  major  prerequisites  for  sueeessful  C2  assessments.  The  first  is  a  elear  and  unambiguous 
statement  of  the  decision  or  issue  to  be  addressed.  The  second  is  a  well  qualified  assessment  team  with 
adequate  time  and  resources.  While  this  may  seem  obvious,  all  too  often  either  one  or  both  of  these  two 
conditions  are  not  met.  In  these  cases,  it  may  be  best  not  to  undertake  an  effort  that  will  be  doomed  to 
failure  from  the  start.  This  extreme  option  can  be  avoided  by  properly  scoping  a  study  to  make  the  effort 
feasible. 

To  begin  with,  every  effort  needs  to  be  made  to  ensure  that  the  assessment  team  understands  the  problem 
or  issue  they  are  to  address.  Any  statement  of  the  problem  will  contain  a  set  of  implicit  assumptions  and 
constraints  that  may  be  known  and  understood  by  the  decisionmaker  but  not  by  members  of  the 
assessment  team.  An  effort  should  be  made  to  make  these  explicit. 

C2  assessments  are  often  undertaken  on  very  “ambitious”  schedules  with  inadequate  resources. 
Experience  has  shown  that  C2  assessments  involve  great  complexity,  difficulty  in  getting  appropriate 
data,  and  the  need  for  significant  sensitivity  analysis  to  deal  with  uncertainty  and  risk.  Therefore, 
plan  conservatively.  This  will  avoid  the  situation  where  the  effort  runs  out  of  time  and  resources  before 
significant  parts  of  the  assessment  can  be  completed  or  the  assessment  team  is  forced  to  not  consider  or 
measure  or  analyse  key,  often  driving  aspects  of  the  problem. 

An  assessment  team  will  rarely  have  all  of  the  expertise  or  experience  needed  to  do  the  job  right. 
The  same  is  true  for  empirical  data  and  the  results  of  other  efforts.  Access  to  the  right  people  and 
information  is  every  bit  as  important  as  having  a  well  articulated  problem  and  adequate  time  and 
resources.  It  is  essential  that  these  needs  are  anticipated  before  the  study  begins  so  that  appropriate 
arrangements  for  access  are  arranged.  Failure  to  accomplish  this  at  the  start  often  results  in  delays, 
cost  overruns,  and  compromises  the  quality  of  the  products.  Additionally,  the  routine  collection  of  data 
during  exercises  and  military  operations  will  facilitate  a  standing  collection  of  data  available  for  future  C2 
assessments. 

The  NATO  Code  of  Best  Practice  for  C2  Assessment  has  proven  itself  to  be  useful  to  both  highly 
experienced  analysts  and  those  without  much  experience.  It  is  recommended  that  decisionmakers  insist 
that  the  assessment  team  review  the  COBP  before  they  develop  their  study  plans  and  inform  the 
decisionmaker  if,  when,  and  why  the  advice  and  or  processes  contained  in  the  COBP  will  not  be  followed. 
Any  deviation  from  the  best  practices  described  in  the  COBP  carry  risks  and  these  risks  need  to  be 
understood  by  decisionmakers  before  they  agree  to  waive  portions  of  the  COBP  for  a  particular  effort. 


Ensure  that  the  problem  is  understood 
Allocate  adequate  time  and  resources  to  the  effort 
Plan  conservatively 

Ensure  study  leadership  has  appropriate  breadth  and  experience 
Ensure  access  to  needed  expertise  and  data 
Mandate  the  use  of  the  COBP 
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Chapter  3  -  OVERVIEW  OF  THE  ASSESSMENT  PROCESS 


Figure  1  depicts  the  C2  Assessment  Process.  This  process  is  iterative.  It  is  applicable  to  any  type  of  C2 
assessment,  regardless  of  the  scope  or  focus  of  the  assessment.  In  fact,  each  of  the  steps  in  the  assessment 
process  will  be  revisited  several  times  during  the  course  of  the  effort. 


A  non-linear, 
iterative  process 


0 

External  Review 

Pixxiuct 

□ 

Pnxess 

Figure  1 :  C2  Assessment  Process. 


In  Problem  Formulation,  the  Assessment  Team  answers  the  question  of  what  is  to  be  addressed  by  the 
effort.  The  Solution  Strategy  addresses  how  this  will  be  accomplished.  The  remainder  of  the  process 
carries  out  the  solution  strategy  and  produces  the  assessment  products.  Problem  Formulation  and 
development  of  the  Solution  Strategy  for  C2  assessments  should  be  an  iterative  process. 

The  decisionmaker  should  schedule  an  initial  review  with  the  assessment  team  to  ensure  that  they  are  on 
the  right  track  after  the  assessment  team  has  gone  through  at  least  one  full  iteration,  specifying  not  only 
what  they  will  do  but  also  thinking  about  each  of  the  remaining  steps  in  the  process  in  some  detail.  Prior  to 
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this  review,  the  assessment  team  should  have  their  study  plan  (this  eonsists  of  both  a  statement  of  the 
formulated  problem  and  their  solution  approaeh)  peer  reviewed. 

It  should  be  expected  that  modifications  will  need  to  be  made  in  either  the  solution  approach  or  in  the 
formulated  problem  as  greater  understanding  is  gained.  The  decisionmaker  must  be  informed  when 
significant  changes  are  made.  Such  changes  may  for  example  be  the  result  of  discovering  that  some  data 
needed  to  support  a  particular  measure  of  merit  (MoM)  is  not  available.  In  such  cases  the  study  plan  needs 
to  be  revised  to  develop  a  surrogate  MoM,  collect  the  appropriate  data,  or  conduct  a  sensitivity  analysis. 

Risks  and  uncertainties  are  an  inherent  part  of  C2  assessments.  These  risks  and  uncertainties  can  not  be 
completely  eliminated,  therefore  they  must  be  managed.  It  should  be  noted  that  each  iteration  ends  with 
a  consideration  of  residual  study  risk.  Decisionmakers  should  expect  that  any  recommendations  to 
significantly  change  the  study  plan  should  be  accompanied  by  an  explanation  of  the  risks  to  study 
objectives,  schedule,  and  costs.  Hence,  the  continuous  involvement  of  the  decisionmaker  ensures  that  the 
study  effort  remains  appropriately  focused  and  that  the  study  team  is  kept  aware  of  decisions  and 
developments  that  infiuence  the  study. 


Involvement  of  decisionmaker  is  key 

Process  must  be  iterative 

Initial  review  needed  to  ensure  proper  start 

Look  for  explicit  treatment  of  risk  and  uncertainty 

Expect  adoption  of  study  plan  as  insight  is  gained 
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Chapter  4  -  MONITORING  THE  ASSESSMENT 


The  first  two  steps  in  the  assessment  proeess  are  Problem  Formulation  and  Solution  Strategy.  The  product 
of  these  two  steps  is  the  study  plan  which  describes  the  what  and  how  of  the  assessment  effort. 

4. 1  PARTICIPATING  IN  PROBLEM  FORMULATION 

Effective  problem  formulation  is  fundamental  to  the  success  of  any  assessment,  but  particularly  C2 
assessment  because  C2  issues  are  often  ill-defined  and  complex.  The  problem  formulation  process 
identifies  the  context  of  the  study  and  aspects  of  the  problem-related  issues  for  assessment.  The  context  of 
the  study  includes  the  geopolitical  environment,  aims  and  objectives  of  the  study,  and  the  decisions  to  be 
supported  by  the  assessment.  Problem-related  aspects  include  the  issues  to  be  addressed,  assumptions, 
high-level  MoMs,  independent  variables  (both  controlled  and  uncontrolled),  and  constraints  on  variables. 
It  must  be  recognised  that  problem  formulation  takes  time  and  must  precede  development  of 
the  assessment  concept  or  selection  of  the  solution  approach  (including  methods  and  tools). 
The  decisionmaker  plays  a  critical  role  in  problem  formulation  by  providing  an  understanding  of  the  real 
problem  to  be  addressed  and  how  study  results  will  support  pending  decisions. 

The  team  should  be  expected  to  quickly  cover  the  whole  problem  and  produce  an  initial  problem 
formulation  to  prevent  premature  narrowing  of  the  assessment  and  to  allow  a  common  understanding 
among  all  team  members.  This  will  identify  the  key  issues  to  be  addressed  and  define  the  context  of  the 
study. 

As  a  result  of  this  process,  the  assessment  team  may  discover  important  issues  related  to  the  problem  that 
need  to  be  discussed  with  the  decisionmaker. 


All  key  variables  should  be  included  in  the  assessment 
Relationships  should  be  known  or  hypothesized 
All  controllable  variables  should  be  identified 
Key  assumptions  should  be  made  explicit 


4.2  MONITORING  SOLUTION  STRATEGY 

A  solution  strategy  consists  of  the  specification  of  a  set  of  sequential  and  parallel  analytical  steps,  often 
involving  several  methodologies  and  tools.  The  solution  strategy  should  begin  with  what  is  known,  and  by 
its  execution,  lead  to  what  the  decisionmaker  desires  to  know  -  insight  into  the  issue(s).  The  solution 
strategy  can  be  simple,  moderately  complicated,  or  extremely  complex. 

The  development  of  a  solution  strategy  is  an  iterative  process  that  should  strike  an  artful  balance  between 
what  the  team  would  like  to  do  and  what  is  possible  to  do,  given  the  state  of  the  art,  available  data,  tools, 
schedule,  and  resources  available.  The  team  first  elaborates  on  the  measures  that  are  to  be  evaluated  in  the 
study.  Using  these  measures  and  consideration  of  human  and  organisational  factors,  a  conceptual  model  of 
the  assessment  is  developed.  The  conceptual  model  is  based  on  the  issues  formulated  in  the  problem 
formulation  and  is  not  driven  by  the  availability  of  tools  and  data.  The  conceptual  model  is  the 
embodiment  of  our  current  understandings  and  may  be  changed  by  the  study  findings. 

Frequently  a  solution  strategy  becomes  complex,  requiring  the  team  to  decompose  the  problem  into  parts, 
each  of  which  requires  assessment  with  its  own  set  of  tools.  Taken  together,  the  solution  strategy  consists 
of  the  MoMs,  relevant  human  and  organisational  factors,  specification  of  scenarios,  data  collection 
requirements,  and  methods  and  tools  to  be  used  in  the  assessment. 
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Potential  problems  should  be  identified  with  contingeney  plans 
Solution  strategy  should  be  clearly  articulated 

Conceptual  model  should  not  be  constrained  by  available  data  or  tools 
Conceptual  model  initially  represents  current  understandings  and  may  evolve 


4.3  APPROVING  THE  STUDY  PLAN 

The  solution  strategy  is  documented  in  a  study  plan  that  links  the  problem  formulation  and  solution 
strategy  together  in  one  plan.  The  study  plan  should  be  complete  and  include:  MoMs,  Human 
Organisational  Factors,  Scenarios,  Methods  and  Tools,  Data  consideration.  Peer  reviews,  in-process 
reviews,  risk  and  uncertainty,  and  products.  The  Study  Plan  should  be  developed  in  an  iterative  fashion, 
applying  guidance  and  feedback  received  from  the  decisionmaker  and  other  stakeholders.  It  is  important 
that  the  Study  Plan  is  peer  reviewed. 

The  study  plan,  presented  by  the  assessment  team  to  the  decisionmaker,  should  be  approved  before 
significant  resources  are  expended.  Often  the  study  plan  is  supported  by  a  study  management  plan  to 
guide,  manage,  and  coordinate  the  effort.  The  study  management  plan  may  have  subordinate  plans, 
to  include  an  analysis  plan,  modelling  and  simulation  plan,  data  collection  plan,  configuration 
management  plan,  quality  assurance  plan,  review  plan,  deliverable  plan,  security  plan,  as  well  as  a  study 
risk  register,  and  glossary. 


Study  plan  should  be  complete 

Study  plan  should  be  realistic  in  terms  of  schedule  and  resources 
And  allow  for  multiple  interations 


4.4  CONDUCTING  IN-PROCESS  REVIEWS 

The  in-process  reviews  are  critical  to  the  success  of  the  effort.  They  should  be  at  key  junctures  including: 
After  initial  problem  formulation.  After  initial  version  of  study  plan.  Prior  to  data  collection.  Between 
collection  and  assessment.  Prior  to  draft  products.  These  are  an  opportunity  for  the  decisionmaker  to  make 
sure  that  the  study,  if  completed  as  planned,  will  answer  the  question  of  interest  and  also  to  make 
appropriate  changes  in  the  study  plan  to  correct  its  deficiencies.  Furthermore,  reviews  help  the 
decisionmaker  to  harmonise  studies  conducted  in  parallel  under  their  aegis  dealing  with  interrelated  issues. 

During  the  initial  review  the  decisionmaker  needs  to  make  sure  that  all  of  the  key  assumptions  related  to 
the  assessment  are  fully  discussed  and  are  appropriate.  In  addition,  these  reviews  should  include  a  full 
discussion  of  study  and  decision  risks. 

A  discussion  of  the  results  of  the  peer  review  and  a  review  of  the  products  that  the  study  will  produce 
should  also  be  done  at  this  time. 


Conduct  reviews  at  key  junctures: 
After  initial  problem  formulation 
After  initial  version  of  study  plan 
Prior  to  data  collection 
Between  collection  and  assessment 
Prior  to  draft  products 
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4.5  ACCEPTING  MEASURES  OF  MERIT  (MoMS) 

MoMs  are  central  to  meeting  the  objectives  of  all  assessments.  Their  development  is  particularly 
challenging  for  C2  assessments,  given  the  nature  of  the  problems  addressed.  It  is  recognised  that  no  single 
measure,  or  class  of  measures,  is  sufficient.  An  orchestrated  set  of  MoMs  is  typically  required  for  C2 
assessments.  The  COBP  has  adopted  the  following  hierarchical  set  of  MoMs: 

•  Measures  of  Policy  Effectiveness  (MoPE)  that  focus  on  policy  or  societal  outcomes; 

•  Measures  of  Force  Effectiveness  (MoFE)  that  focus  on  how  a  force  performs  its  mission  or  the 
degree  to  which  it  meets  its  objectives; 

•  Measures  of  C2  Effectiveness  (MoCE)  that  focus  on  the  impact  of  C2  systems  within  the 
operational  context; 

•  Measures  of  Performance  (MoP)  that  focus  on  internal  system  structure,  characteristics  and 
behaviour;  and 

•  Dimensional  Parameters  (DP)  that  focus  on  the  properties  or  characteristics  inherent  in  the  C2 
system. 

A  good  C2  assessment  will  establish  relationships  among  the  measures  used  in  the  assessment. 
Establishing  these  relationships  between  types  of  measures,  however,  is  among  the  most  challenging  parts 
of  any  assessment.  It  is  important  to  recognise  that  even  approximate  relationships  among  the  right  MoMs 
are  far  preferable  to  “precise”  relationships  between  MoMs  that  do  not  adequately  reflect  the  key  issues 
at  hand.  Criteria  have  been  developed  that  help  to  ensure  that  the  measures  that  are  selected  are  both  valid 
and  reliable.  For  a  measure  to  be  considered  valid,  it  must  be  mission  oriented,  realistic,  appropriate, 
inclusive,  discriminatory,  meaningful,  simple,  relevant,  and  generalisable.  For  a  measure  to  be 
considered  reliable,  it  must  be  discriminatory,  measurable/observable,  quantitative,  objective,  sensitive, 
and  consistent.  A  proposed  measure  may  be  reliable  but  not  valid,  or  it  may  be  valid  but  not  reliable. 
The  decisionmaker  should  insist  that,  to  the  extent  possible,  the  measures  to  be  used  are  valid,  reliable, 
and,  if  calculated,  will  provide  desired  information.  The  risk  assessment  should  characterise  the  degree  to 
which  the  MoMs  may  not  be  valid  and  reliable  and  describe  the  mitigating  actions  taken. 


Ensure  MoMs  are  valid,  measured  reliably,  credible  and  complete 


4.6  CONSIDERING  HUMAN  AND  ORGANISATIONAL  FACTORS 

The  human  dimension  largely  distinguishes  C2  assessment  from  other  military  operations  assessment. 
C2  assessment  must  deal  with  distributed  teams  including  military,  interagency,  coalition  and  other 
non-state  actors  operating  under  stress  and  their  varying  decisionmaking  behaviours.  In  operations  other 
than  war,  particular  attention  must  be  paid  to  behaviour  of  and  interaction  with  non-military  organisations, 
political  groups,  and  amorphous  groups  such  as  crowds  and  refugees. 

Thus,  the  formulation  of  the  problem  and  the  development  of  solution  strategies  cannot  be  completed 
without  explicit  consideration  of  both  human  and  organisational  issues. 

Human  performance  affects  behaviour  and  vice  versa.  Human  performance  depends  on  psycho- 
physiological,  and  cognitive  variables  such  as  stress,  fatigue,  sleep  deprivation,  hunger,  alertness, 
personality,  and  predisposition,  as  well  as  on  ergonomic  and  external  factors.  Individual  and  group 
behaviours  are  also  the  result  of  social  interaction  impacted  by  fear,  morale,  value  systems,  culture, 
education,  and  religious  backgrounds  of  individuals.  When  human  performance  and/or  behaviour  is  at 
issue,  parameters  and/or  models  will  need  to  reflect  those  issues.  Particularly  critical  is  how  human 
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decisionmaking  is  addressed  in  the  assessment,  and/or  in  models  used  in  the  assessment.  The  human 
factors  mentioned  above,  as  well  as  the  command  style,  risk  style,  and  other  command  attributes  that 
impact  human  decisionmaking,  must  be  accounted  for  in  the  modelling  and  assessment.  The 
decisionmaker  can  be  invaluable  in  this  regard  if  he/she  is  also  the  subject  of  assessment  by  making  sure 
the  assessment  team  understands  their  approach  to  decisionmaking. 

There  is  a  strong  direct  link  between  human  and  organisational  issues.  Properly  done,  organisational 
design  reflects  the  interaction  of  tasks  to  be  done,  the  people  available  to  perform  them,  and  the  systems  or 
tools  that  support  them.  Because  of  the  need  for  co-evolution  organisational  design  is  often  a  key 
independent  variable.  Organisations,  then,  depend  on  the  capabilities,  training,  and  experience  of  the 
people  in  the  C2  system.  Organisational  factors  must  be  addressed  as  part  of  most  C2  analyses. 
Organisational  factors  include  structural  (e.g.  number  of  echelons,  span  of  control),  functional 
(e.g.  distribution  of  functions,  information,  and  authority),  and  capacity  (e.g.  personnel,  communications) 
factors.  The  large  number  of  organisational  variables  that  may  be  relevant  to  C2  assessment  must  be 
approached  carefully  and  systematically.  When  possible,  organisation  theory  expertise  should  be  brought 
into  the  team.  The  assessment  team  will  be  challenged  to  identify  factors  that  are  particularly  relevant  to 
their  C2  assessment,  and  to  identify  and  employ  appropriate  tools. 


Explicit  consideration  of  human/organisation  performance 
Inclusion  of  appropriate  expertise  on  the  team 


4.7  APPROVING  SCENARIOS 

The  selection  of  a  proper  set  of  scenarios  is  critical  to  the  assessment.  Scenarios  consist  of  four  elements  - 
a  context  (e.g.  a  characterisation  of  a  geopolitical  situation),  the  participants  (e.g.  intentions,  capabilities  of 
friendly,  hostile,  neutral),  the  environment  (e.g.  natural  -  weather  and  man  made  -  mines),  and  the 
evolution  of  events  in  time.  Scenarios  can  be  approved  or  unapproved.  Some  are  operational  scenarios, 
meaning  they  contain  additional  details  and  may  exist  in  a  model.  Sometimes  smaller  scenarios,  called 
vignettes  are  developed  for  analytic  use. 

In  C2  assessments,  the  purpose  of  scenarios  is  to  ensure  that  the  assessment  is  informed  by  decisionmaker 
planning  assumptions  and  the  appropriate  range  of  opportunities  to  observe  the  relevant  variables  and  their 
interrelationships.  Although  the  ideal  would  be  for  the  assessment  to  be  scenario  independent,  rarefy  does 
this  happen  due  to  the  breadth  and  complexity  of  C2.  Therefore,  scenarios  must  be  considered  throughout 
the  assessment  process,  especially  during  problem  formulation.  In  essence,  the  role  of  scenarios  is  to 
define  a  set  of  conditions  and  restrictions  to  enable  credible  assessment  as  well  as  to  create  a  structure 
within  which  the  results  of  the  assessment  can  be  understood  and  interpreted. 

Scenarios  must  be  designed  or  selected  to  address  C2  under  a  broad  range  of  circumstances. 
C2  assessments  need  to  use  multiple  scenarios,  no  single  scenario  is  sufficient. 

For  C2  assessment,  scenarios  should  reflect  the  C2  organisation,  processes,  and  systems  relevant  to  the 
assessment.  In  selecting  scenarios  for  C2  assessment,  the  analyst  should  ensure  that  the  scenarios  reflect 
the  factors  that  have  significant  impact  on  C2,  stress  C2  issues,  are  credible  to  the  military,  are  credible  in 
terms  of  civil-military  objectives,  and  will  facilitate  the  study  design  process.  Note  that  this  implies  a 
broader  selection  of  scenarios  than  is  normally  contained  within  current  sets  of  “approved”  scenarios. 
Due  to  their  critical  importance  to  the  study,  the  decisionmaker  must  pay  particular  attention  to  the  design 
and  selection  of  scenarios. 
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Ensure: 

Multiple  scenarios 

The  set  of  scenarios  cover  the  range  of  relevant  situations 


4.8  UNDERSTANDING  THE  METHODS  AND  TOOLS 

There  is  a  broad  range  of  methods  and  tools  available  to  be  applied  to  C2  assessment.  The  methods  and 
tools  fall  into  one  or  more  of  four  categories  -  data  collection/generation  tools  (e.g.  simulations,  exercises, 
experiments,  expert  elicitation,  real  world  operations),  data  organisation  tools  (e.g.  causal  mapping,  multi¬ 
criteria  decision  analysis,  neural  nets),  “solving”  tools  (e.g.  mathematical  analysis,  linear  programming, 
goal  programming),  and  support  tools  (e.g.  data  analysis,  databases,  checklists,  spreadsheets).  Although 
the  focus  of  recent  past  research  has  been  on  the  development  of  computer  simulations,  virtually  any  of 
the  analytic  tools  in  the  analyst’s  inventory  could  potentially  be  applied  to  C2  assessment. 

The  development  or  selection  of  tools  for  C2  assessment  should  be  based  on  evaluation  of  candidates 
against  functionality-based  and  performance-based  selection  criteria.  Functionality-based  selection  include 
resolution/detail,  completeness/scope,  the  functionality  coverage  provided,  the  explicitness  of  entity 
representation,  the  ability  to  generate  appropriate  MoMs,  and  whether  the  tool  has  been  verified,  validated, 
and  accredited  (VV&A)  for  the  intended  use.  Performance-based  selection  criteria  include  responsiveness, 
simplicity,  time  to  prepare/use,  data  availability,  interoperability  with  other  tools,  resources  required,  and 
credibility  with  customers  and  users.  Decisionmakers  should  refer  issues  regarding  the  selection  of 
methods  and  tools  to  peer  review. 

Although  the  assessment  team  will  employ  any  tool  that  assists  in  addressing  study  issues,  credible 
computer  simulations  are  what  most  analysts  seek.  The  development  of  C2  simulations,  especially  those 
that  link  C2  to  force  effectiveness,  has  been  the  subject  of  much  research  recently.  Decisionmakers  should 
be  aware  that,  although  significant  progress  has  been  made,  many  challenges  still  exist  in  the  modelling  of 
C2.  Among  these  challenges  are  representation  of  human  behaviour,  linking/federating  models, 
representing  adversarial  entities,  dealing  with  uncertainty  in  model  representations,  conduct  of  sensitivity 
analysis  with  models,  and  the  VV&A  process. 

The  assessment  team  should  seek  approval  from  the  decisionmaker  for  the  models  that  will  be  employed 
as  part  of  the  assessment.  Decisionmakers  should  ask  how  C2  is  represented  in  proposed  models  and  how 
model  outputs  will  provide  the  data  necessary  to  address  the  C2  issues  under  study. 


Study  needs  should  drive  tool  selection,  not  vice  versa 
Seek  clear  rationale  for  tool  selection 
Multiple  tool  are  normally  needed 

Demand  explicit  linkage  from  methods  and  tools  to  study  issues 


4.9  COMPREHENDING  THE  DATA 

The  role  and  importance  of  data  in  C2  assessments  is  underestimated  by  many.  The  ability  to  determine 
what  data  are  needed  and  the  ability  to  collect  these  data  in  fact  determine  in  large  part  the  solution 
strategy  (an  example  of  the  importance  of  thinking  of  the  assessment  process  as  iterative  in  nature). 
There  may  be  many  types  of  data  required,  to  include  scenario  data,  human  performance  data,  and  systems 
performance  data.  Decisionmakers  and  analysts  prefer  data  to  be  sharp,  certain,  complete,  and  consistent, 
but  recognise  this  will  not  always  be  the  case.  For  C2  assessment,  it  is  particularly  difficult  to  meet  these 
criteria. 
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Hence,  not  all  C2  related  data  required/desired  by  the  analysts  will  be  readily  available.  Some  of  it  will 
have  to  be  aggregated  or  derived  from  other  sources,  or  perhaps  generated  from  original  sources. 
C2  related  data  can  be  obtained  from  a  variety  of  sources  to  include  official  sources  (e.g.  military 
databases),  open  sources  (e.g.  Internet),  legacy  studies,  and  subject  matter  experts.  The  assessment  team 
must  understand  what  data  are  needed  and  in  what  form,  who  owns  the  data  if  it  already  exists,  and  the 
costs  involved  in  buying,  collecting,  or  generating  required  data.  These  costs  can  be  significant. 
The  decisionmaker  can  help  the  assessment  team  by  assisting  in  identifying  available  or  potential  data 
sources.  Furthermore,  the  quality  of  the  study  results  is  influenced  by  the  quality  of  data  used  as  much  as 
by  the  selection  of  respective  methods  and  tools. 


Articulate  the  assumptions  related  to  collection 

Understand  date  anomalies  and  adjustments  made  to  data 

Understand  data  sources 

Look  for  data  distributions,  not  just  averages 

Help  the  assessment  team  get  the  data  they  need 


4.10  UNDERSTANDING  RISKS  AND  UNCERTAINTIES 

There  are  risks  associated  with  the  decision  at  hand  and  there  are  risks  related  to  the  assessment  process. 
Failure  to  deal  effectively  with  both  of  these  types  of  risk  will  jeopardise  study  goals. 

Risk  is  commonly  defined  as  the  possibility  of  suffering  harm  or  loss,  to  include  opportunity  loss. 
Risk  often  has  a  negative  connotation,  yet  “taking  risks”  can  also  be  positive.  Uncertainty  can  be  defined 
as  an  inability  to  determine  a  variable  value  or  system  state  or  predict  its  future  evolution.  Uncertainty  is 
inherent  in  risk.  Risk  and  uncertainty  are  especially  prevalent  in  C2  and  in  C2  assessment.  C2  issues  are 
inherently  complex  and  have  many  interacting  factors.  Additionally,  C2  assessment  is  effected  by 
uncertainties  in  scenarios,  data,  and  models. 

Risk  can  be  dealt  with  in  C2  assessments  by  either  reducing  the  uncertainty  that  underlies  the  risk, 
by  embracing  and  accounting  for  residual  uncertainties,  or  by  communicating  the  risk  involved  and 
adopting  a  decision  strategy  that  mitigates  the  risks.  Uncertainty  can  be  reduced  in  C2  assessment  in  a 
number  of  ways,  to  include  conducting  certification  of  data,  conducting  proper  VV&A  of  models,  and  in 
conducting  sensitivity  analysis.  The  assessment  team  should  make  a  serious  effort  to  illuminate  the  risks 
and  to  conduct  an  explicit  risk-based  assessment  (consideration  of  multiple  potential  solutions). 
With  problems  involving  human  decisionmaking  the  analyst  must  be  aware  of  the  diversity  of  courses  of 
action  that  are  possible  as  a  scenario  evolves.  The  analyst  should  also  attempt  not  to  unduly  bound  the 
problem  during  problem  formulation  to  reduce  risk.  The  decisionmaker  can  help  the  analyst  with  these 
efforts  by  helping  to  identify  risks  and  uncertainties  and  by  discussing  his  or  her  own  approach  to  risk  and 
uncertainty  regarding  the  study  issues  and  the  decisions  to  be  made  based  on  the  assessment. 


Failure  to  adequately  deal  with  risk  jeopardises  study  goals 
Sensitivity  analyses  should  be  provided  and  explained 
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Chapter  5  -  REVIEWING  THE  PRODUCTS 


Assessment  produets  inelude  the  study  plan,  periodie  status/progress  reviews,  and  the  final  report. 
The  deeisionmaker  has  an  important  role  in  eaeh.  The  study  plan  should  be  presented  at  the  initial  review 
and  the  decisionmaker  should  approve  it.  The  study  plan  should  serve  as  a  shared  understanding  between 
the  decisionmaker  and  among  the  study  participants.  However  the  iterative  nature  of  good  C2  assessments 
means  the  study  plan  should  be  flexible. 

Periodic  reviews  should  be  chaired  by  the  decisionmaker.  The  study  plan  should  go  through  a  peer  review 
process  prior  to  finalisation.  In  addition  to  these  products,  there  are  a  number  of  other  products  that  best 
practice  demands  be  produced  and  maintained  during  the  course  of  the  study.  These  include  a  project 
journal,  a  study  design  and  implementation  plan,  a  data  collection  plan,  a  data  analysis  plan,  and  a  risk 
register.  All  products  should  be  archiveable  and  readily  accessible  by  the  community,  within  the 
constraints  of  security.  It  is  recommended  practice  that  a  version  is  prepared  at  the  lowest  classification 
possible  to  allow  for  the  widest  distribution  possible  and  reuse  in  the  assessment  community. 


The  decisionmaker  should  be  able  to  brief  the  study  and  explain  its  results 
Data  should  be  saved  in  a  form  amenable  to  re-use 
Study  products  should  be  peer  reviewed 
A  plan  should  be  developed  for  archiving  and  dissemination 
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Annex  A  -  DECISIONMAKER’S  TEMPORAL  QUESTION  LIST 

This  annex  contains  key  questions  that  a  decisionmaker  should  ask  the  C2  assessment  team. 
These  questions  are  organised  temporally  according  to  the  following  phases  of  a  study  (prior  to  the  study, 
at  initial  review,  after  first  iteration,  and  at  final  report). 

Prior  to  the  Study: 

•  Do  you  understand  what  decisions(s)  I  have  to  make,  when  I  have  to  make  them,  and  the  context 
within  which  the  decision(s)  will  be  made? 

•  Do  you  need  any  information  or  authorisation  from  me? 

•  Who  will  be  on  the  study  team? 

•  Are  there  adequate  skills,  experience  present  in  the  team? 

•  For  OOTW  studies,  in  particular,  are  there  adequate  social  scientific  skills  in  the  team? 

•  Who  are  the  key  organisations/individuals  with  whom  you  plan  to  interact  (e.g.,  stakeholders,  data 
providers,  review  team)? 

•  Have  you  coordinated  the  Terms  of  Reference  with  them? 

•  How  do  you  plan  to  interact  with  them? 

•  Particularly  for  OOTW  studies,  how  will  you  acquire  the  requisite  knowledge  of  the 
culture/historical  context? 

•  How  will  you  undertake  problem  formulation?  e.g., 

•  What  products  will  you  review/mine? 

•  What  methods  and  tools  are  applicable? 

•  When  will  key  events  occur?  (e.g.,  reviews,  production  of  interim  products) 

At  the  Initial  Review: 

•  What  do  you  perceive  the  “real”  issues  to  be? 

•  What  assumptions  do  you  plan  to  make  to  scope  the  effort? 

•  What  do  you  plan  to  use  for 

•  High  level  MoMs? 

•  Scenarios  of  interest? 

•  Have  you  identified  any  additional  organisations/individuals  with  whom  you  plan  to  coordinate? 

•  How  do  you  plan  to  attack  the  problem? 

•  What  methodology  will  you  employ? 

•  How  will  you  treat  the  diverse  aspects  of  a  mission  capability  package  (including  concept  of 
operations,  organisation,  doctrine,  C2  approach,  systems,  personnel,  and  facilities,  in  other 
words  everything  needed  to  field  a  real  capability)? 

•  How  do  you  plan  to  address  organisation/human  issues? 

•  What  specific  methods  and  tools  will  you  employ?  Why  do  you  think  they  are  appropriate? 

•  What  data  will  you  employ?  Where  will  you  get  them?  Why  do  you  think  they  are  appropriate? 
How  do  you  plan  to  make  the  data  accessible  to  others?  How  do  you  plan  to  depict  the  results  of 
the  study? 
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At  First  Iteration: 

•  What  specific  MoMs  were  selected?  What  relationships  were  established  among  the  MoMs? 

•  What  range  of  scenarios  were  selected?  Why? 

•  What  plans  do  you  have  to  illuminate  uncertainty/sensitivity? 

•  What  feedback  did  you  receive  from  the  independent  review  team?  What  steps  did  you  take  to 
respond  to  it? 

•  What  do  you  plan  to  do  on  subsequent  iterations? 

•  Use  additional  tools? 

•  Consider  additional  scenarios,  assumptions? 

•  Modify  assessment  boundaries? 

At  Final  Report: 

•  What  are  the  major  findings,  recommendations? 

•  What  are  the  key  points  of  uncertainty/sensitivity? 

•  What  issues  were  not  addressed  that  should  be  treated  in  subsequent  assessments? 

•  What  key  lessons  did  you  learn  with  respect  to  methods,  tools,  and  data? 

•  What  steps  are  you  going  to  take  to  disseminate  key  insights,  products  to  the  broader  community? 

•  Are  there  any  voids/issues  that  warrant  further  research? 
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